Emulate the decrypt, don't reverse it

Some binaries protect their pointers. A global that should hold an object list holds a scrambled 64-bit value instead, and every read site carries its own inline copy of the unscramble: a few rotations, an xor with a constant, an add, sometimes a mask. Across a large image there are thousands of these sites, each one slightly different because the mutation engine rewrote it.

For two weeks I tried to recover the formula by parsing the instructions. Then I gave up on parsing and ran the code instead. That took an afternoon and gave me all four formulas exactly.

Why the parser lost

A linear parser walks the instructions at a site and pattern-matches them into an expression. It works on textbook output. It failed here for four reasons, and none of them are exotic.

The constants were not there. The compiler had folded two of the immediates into other operations, so the xor constant I was looking for did not exist as bytes anywhere in the site.

The rotations were not rotations. ror rax, 52 had been lowered to shr, shl, or over two registers, and my parser counted 49 bits instead of 52 because one of the shifts was applied to a partial width.

The branches were fake. After the xor, a jp or jnp branched on the parity of the value into two paths that compute the same thing. Opaque predicates, and a parser that follows one path sees half the operations.

And the site was not where the work started. The compiler loaded the 64-bit constants into registers before the global load, so a parser that begins at the load sees registers with contents it cannot explain.

Run it, then ask where the value went

The emulator runs the real code from the real image. A CPU emulator executes each instruction; a disassembler decodes it in parallel so I know what happened; and an expression tree follows one source cell, the scrambled value V, through registers and stack spills until it reaches a sink.

A sink is one of three things: the value is dereferenced, the value is passed as a call argument, or the function returns at call depth zero. At the sink, the expression tree is the formula.

site 0x1401a3c50
  V = [rip+0x2a1f8b0]
  t1 = ror(V, 52)
  t2 = t1 ^ 0x9e3779b97f4a7c15
  t3 = t2 - [rip+0x2a1f8b8]
  sink: mov rax, [rdx + t3]      ; deref, displacement 0x0 = field offset

Checking is the important part. For each site, evaluate the tree on four random inputs and compare against the concrete register at the sink. A site whose tree does not reproduce its own register is discarded. Then take the majority over all sites, and compare that against one formula verified live against the running process. Only then is it frozen as an expectation for the next build.

The four traps

Each cost one run.

Constants before the site. Fixed by walking backwards from the site over “safe” instructions, mov, lea, arithmetic, reads that are only rip- or rsp-relative, for up to twelve instructions, and starting emulation there. Starting at the load itself shows canary values in the registers.

The last add is in the addressing mode. mov rax, [rdx+rax] is a sink whose address is base + index*scale + disp. The add is not an instruction, it is the operand, and the displacement is the field offset. The tracker has to read operands, not only opcodes.

Helpers that return. Some sites call a helper that manipulates the value through a pointer and comes back. A ret inside the helper is not a sink. Only a ret at call depth zero ends the trace.

The presence mask from the environment. One family used and rcx, rax with a mask built from registers that were never set in the trace, through neg, movsxd and sbb. Registers start symbolic, flags are tracked, and any value derived from a partial width or a flag becomes an opaque environment constant whose value is read from the register after execution. An and with an environment operand is split off as a mask; any other environment-dependent step discards the site. Better to lose a site than to accept a formula that depends on state I cannot see.

Live memory as the backend

The second half of this is where the pages come from. A static dump is fine for code, but the interesting routines read data that only exists while the process runs: the object behind the pointer, the table behind the object.

So the emulator has a live backend. A small server sits on the driver and answers page requests: 4 KiB per request, with a status flag instead of zeros when a page is unreadable. The emulator maps every page on first touch, whether the touch is an instruction fetch, a data read, or a read from my own script. --record writes every touched page and the routine’s arguments to a bundle; --bundle replays that without the process. An expectation is measured on a dump or a bundle, never live, so the test suite does not depend on a game being open.

The proof of the plumbing was deliberately boring: an ntdll dump as the image, kernel32!lstrlenA fetched live out of explorer.exe, three pages pulled, rax = 11. Replay from the bundle gives the same. Without a source it faults, which is the correct answer.

Where this generalises

The same machinery answers a different question on a different engine: a script VM whose classes exist only in the heap, never in the image. Property access there is not a field at an offset, it is a getter. Three sessions went looking for a slot that does not exist. The answer was in the getter’s code, and the emulator can run a getter against the live object and tell you what it returns, with a trace of every address it touched on the way. That trace is the field map, taken from the running process instead of guessed from a hexdump.

The general rule: when the code is the specification, execute the specification. Parsing is reconstructing what the compiler did; emulation is letting the CPU do it and watching.