IDA Pro Tutorial: Unpacking Obfuscated Binary Using IDA Pro Debugger

The following flags are checked out as well:

• Rename DLL sections
• Create import segment

The subroutine reveals the unresolvable jmp call to nowhere, as marked in red.

Now, we are loading the binary again with the same function using “Manual Load” and unchecking “Create segment section” to avoid at the segment error.

Now, the jmp call leads to the function entitled “start_0.” 

Notably, the jmp call is the first instruction of this binary. In fact, it abused the header conventions by loading the code in these sections.

We see that the binary loads a pointer into ESI which gets immediately copied to EBX:

HEADER:00400158                 mov     esi, offset off_40601C
HEADER:0040015D                 mov     ebx, esi
HEADER:0040015F                 lodsd
HEADER:00400160                 lodsd
HEADER:00400161                 push    eax
HEADER:00400162                 lodsd
*Tip: Ctrl + O converts hexadecimals to labels

Soon, the value of EBX is used to call a subroutine:
HEADER:00400169            call    dword ptr [ebx]

At 0x40601C, the pointer offset off_40601C, or 40601Ch (in hexadecimals when pressed Ctrl+O for conversion) points to the memory location at 0x400130. To get to this point, we undefine the array (hotkey U), go back to the pointer (hotkey Esc) and follow the pointer again.

While the function is now on screen, we only have half of it. It seems that the person who wrote that program wanted it to obfuscate it and separated the function into several pieces.

*Tip: There is a special command to help us: the command to generate flow charts of functions in IDA, it’s hotkey is F12.

A quick glance at the flow chart reveals that there is only one exit from the function at its “ret” instruction ()
First press F2 to create a breakpoint, then right click and select “edit breakpoint” to change it to a hardware breakpoint on the “execution” event.

​After having set the breakpoint, we start the debugger by pressing F9. When we reach the breakpoint, the program will be unpacked into the ‘MEW’ segment. We jump there and convert everything to code (the fastest way to do this is to press F7 at the breakpoint)

There are two things to be saved because they will disappear when the debugger stops: the memory contents and the imported function names. The memory contents can be saved by using the following 4-line script:
auto fp, ea;
fp = fopen(“bin”, “wb”);
for ( ea=0x401000; ea < 0x406000; ea++ )
  fputc(Byte(ea), fp);

If we want to know the name of the called function, we press Enter several times to follow the links and finally get the name:
kernel32.dll:77E7AD86
kernel32.dll:77E7AD86 kernel32_GetModuleHandleA:              ; CODE XREF: sub_4012DCj
kernel32.dll:77E7AD86                                         ; DATA XREF: MEW:off_402000o
kernel32.dll:77E7AD86 cmp     dword ptr [esp+4], 0

We have to copy the function names before that:
auto ea, name;
for (ea = 0x401270; ea<0x4012e2; ea =ea+6 )
{
  name = Name(Dword(Dfirst(ea)));               /* get name */
  name = substr(name, strstr(name, “_”)+1, -1); /* drop the prefix */
  MakeName(ea, name);
}

Now that we have run those scripts, we may stop the debugger (press Ctrl-F2) and copy back the memory contents. The “Load additional binary file” command in the File, Load menu is the way to go:

Note that it is not necessary to create a segment, it already exists (clear the “create segments” flag). Also, the address is specified in paragraphs, i.e. it is shifted to the right by 4.

Load the file, press P at 0x401000 and, you have a nice listing: