Callfuscated: Reverse Engineering a Stack-Based VM Protected by MBA, Opaque Predicates and Call-Based Jumps

The HackTheBox challenge Callfuscated presents one of the most layered crackmes published on the platform, combining virtualization, Mixed Boolean-Arithmetic (MBA), opaque predicates and junk instructions inside a single binary.

Initial analysis shows hundreds of functions that appear to receive arguments through mysterious arg_* variables. Disassembly reveals that every call is immediately followed by a pop r8 instruction; the return address is discarded and only r8 is clobbered. Unicorn emulation of the entire binary confirms that r8 is never subsequently read, proving that each call-pop pair is a hand-written jump.

A lightweight Python script using Capstone walks the code, replaces the call opcode (0xE8) with jmp (0xE9) and overwrites the pop r8 bytes with NOPs. After patching, Ghidra produces a readable main function containing the characteristic dispatch loop of a stack-based virtual machine.

Each handler invokes a separate function protected by MBA. The author employs Triton to symbolically execute these routines and extract an AST for the return value in RAX. The resulting expressions, sometimes exceeding 7 000 characters, are fed to the CoBRA MBA solver, which reduces them to the original operations: addition, subtraction, multiplication, bitwise AND, OR and XOR. Two handlers that receive random values as arguments simplify to expressions that ignore those arguments, confirming the presence of opaque predicates.

With opcode semantics recovered, the author uses Unicorn again to dump the bytecode at the point where it has been fully initialized. A small C emitter translates the bytecode into ordinary C statements that operate on a 256-element stack. The generated file is compiled with optimizations and loaded into a decompiler, yielding a clean routine whose final comparison against zero determines whether the flag is correct.