oalabs

Tracing The Pain Away Module 2 - Debugger Basics

Added 2024-08-29 14:00:06 +0000 UTC

Introduction to API tracing with the debugger.

References

anti-debug.checkpoint.com
Practical Calling Conventions Tutorials
https://www.patreon.com/collection/155851?view=expanded
X64dbg string formatting reference
help.x64dbg.com/en/latest/introduction/Formatting.html
Tracing Memory With X64dbg and Advanced Breakpoints
https://www.patreon.com/posts/tracing-memory-107586165

Lab 1 - Debugging

This lab introduces the concept of using simple API monitoring with a debugger to gain insight into VM operations. You are provided with a reference binary which can be used to understand the code that is protected by the VM, and you are provided with a VM protected version of the reference binary with a small modification.

Debugging VMs: Reference Binary

The lab1.exe reference binary uses the API call GetFileAttributesA to check for the presence of a file. If the file exists you pass the check, if the file does not exist the check fails. The goal is to identify the file name and path.

Open lab1.exe in IDA and locate the call to GetFileAttributesA. Observe the file path that is checked.
Open lab1.exe in x64dbg and run until the binary entrypoint.
In the Symbols tab locate the function GetFileAttributesA in the kernel32 module and place a breakpoint on it.
Run the debugger until your breakpoint is hit and observe the API arguments on the stack. Locate the file path.

You can use MSDN as a reference for locating the arguments
https://learn.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-getfileattributesa

Debugging VMs: VM Binary

The lab1vm.exe binary is an variant of the reference binary that has been protected with VMProtect. Specifically the function that performs the file check has been virtualized.

Open lab1vm.exe in IDA and observe that the function performing the file check can no longer be statically analyzed.

Your goal is to use x64dbg with the same API tracing technique demonstrated on the reference binary to recover the file path checked by the protected binary.