187,064 Instructions for One Flag: Reverse Engineering HTB Callfuscated Insane Challenge

The HackTheBox challenge Callfuscated presents a stripped 64-bit ELF binary that requests a password and prints either Correct or Incorrect flag. Inside the binary lies a heavily obfuscated verification routine built from four complementary techniques that survive static analysis but collapse under concrete execution.

The Obfuscation Arsenal

The protection consists of a custom VM whose bytecode resides in a 586-element array called program[]. A dispatcher reads the current program counter, fetches an opcode, and dispatches to handlers. Every arithmetic primitive is further expanded with Mixed Boolean-Arithmetic expressions. Opaque predicates double the control-flow graph with branches whose outcome is known at compile time. Finally, genuine instructions are wrapped inside thousands of call gadgets of the form pop r8; <insn>; call next, producing the 187,064-instruction count mentioned in the title.

Dynamic Reconnaissance

Because all four layers are execution-dependent, the author recorded a full instruction trace using ptrace with PTRACE_SINGLESTEP on a known input. The resulting log contained 187,064 lines together with register values after each instruction inside the image range 0x401000–0x40d000. A separate memory dumper extracted the VM program array and stack frames directly from the running process.

Building a Faithful Emulator

The trace was replayed inside a custom x86-64 emulator written in Python that parsed objdump output and implemented handlers for the thirty opcodes actually encountered. Two notable bugs were corrected during validation:

• rand() is invoked from four distinct call sites (0x409235, 0x40ad5b, 0x40af37, 0x40b10b) for a total of 192 calls; each return address must be calculated as call-site + 5 rather than assuming a single fixed location.
• Operand-size detection used a naïve substring check that incorrectly treated “DWORD PTR” as containing “WORD”, truncating 32- and 64-bit memory accesses.

After these fixes the emulator matched the original trace line-by-line and reached the final instruction at address 0x40b537 with eax equal to 0xffffffff, exactly as the real binary behaves on an incorrect password.

Recovering the VM Semantics

With a working emulator the author dumped the data stack after every dispatch and decoded the 586 VM instructions. The bytecode implements a simple stack machine whose relevant operations are:

• PUSH imm – push constant
• H2 – addition (used to compute input buffer address 0x40f080)
• DEREF – read input byte
• H5 – multiply accumulator by 256
• H7 – add next byte
• G8 / G3 – XOR with per-group constants

Consequently every four input characters are accumulated into a 32-bit big-endian word. Eight such words are XORed with the following constant pairs:

• (0x0915033a, 0x41414141)
• (0x427d7872, 0x11111111)
• (0x30310a00, 0x55555555)
• (0x2a052e32, 0x5a5a5a5a)
• (0xcff5ecdf, 0xaaaaaaaa)
• (0x1914031e, 0x77777777)
• (0xf6f7c6ad, 0x99999999)
• (0x6c6a524e, 0x33333333)

The resulting 32-bit values must all be zero for the password to be accepted. Solving each equation yields the flag bytes directly: HTB{******_**_***_********_*_**}.

Conclusion

The exercise demonstrates that even an extreme combination of VM-based dispatch, MBA, opaque predicates and call obfuscation remains vulnerable once execution is recorded and replayed. All four techniques ultimately depend on concrete runtime values that a faithful emulator can capture and simplify.