Reversing MD5 Hash Function from 2500-Layer Neural Network in Jane Street CTF Puzzle

A Jane Street puzzle released in February last year presented participants with a complete PyTorch model in pickle format containing roughly 2500 linear layers. The model, stored in model.pt, produced an output of 0 for almost every input, including the example strings “vegetable dog”. The challenge required discovering an input that would make the network return a non-zero value without relying on gradient descent or exhaustive search.

Initial inspection of the output layers

Participant Alex began by examining the final two linear layers. The last layer was a 48×1 weight matrix clearly divided into three equal sections, while the preceding layer contained three copies of identical weights together with a repeating 16-byte bias pattern incremented by one each time. This structure indicated that the network maintained three versions of a 16-byte vector v and compared them against a secret target x using the combination v, v+1, v+2. The final layer applied weights 1, −2 and 1, so that only an exact match across all 16 bytes produced a positive result after the bias of −15 was applied.

Network simplification and constraint solving

The remaining 2500 layers formed a directed acyclic graph of integer operations. Alex modeled the entire network as an integer linear program, introducing binary variables to encode the behavior of each ReLU activation. After repeated simplifications—merging identity mappings, removing redundant ReLUs on strictly positive paths, and collapsing duplicate neurons—the problem size dropped from approximately two million nodes to 75 000. Even then, both an ILP solver and a subsequent SAT encoding with 200 000 variables failed to finish within practical time limits.

Discovery of the MD5 core

Plotting layer widths revealed 32 identical periods of length 48. Consulting common cryptographic primitives, Alex recognized the structure of the MD5 compression function. Manual verification confirmed that intermediate activations matched MD5 round constants and state variables, while other hash functions did not. The target 128-bit value was already visible in the bias of the penultimate layer, reducing the original problem to finding a preimage under that specific MD5 hash.

Unintentional length-encoding bug

Further reverse engineering exposed a flaw in the first seven layers responsible for encoding message length in little-endian format. When the input length reached or exceeded 256 bits, the network stored the raw integer 256 instead of the correct four-byte representation. This error affected only a subset of MD5 blocks yet prevented correct hashing for any input longer than 32 bytes. The bug was later confirmed by the puzzle authors to be unintentional.

Final solution

Once the algorithm and the target hash were known, the remaining task was a modest brute-force search over two-word English phrases. A larger word list quickly yielded the correct input that satisfied the hidden MD5 value. The puzzle demonstrated that a carefully constructed neural network can embed a non-differentiable cryptographic routine while still remaining solvable through systematic simplification and algorithmic insight.