AI Agents Codex and Grok Generate Passing Tests That Fail to Verify Cookie Signatures and Security Logic

An experienced developer using Codex and Grok to write production code and accompanying tests has documented a series of cases where tests passed despite failing to validate core security mechanisms.

In a Go service that issues signed cookies of the form base64(payload).base64(hmac), the agent produced a tamper-rejection test that replaced the first character of the encoded string. Because the payload is a JSON object encoded in base64 and therefore begins with “e”, the mutation broke JSON parsing. The Decode function returned ErrInvalidSignature for any failure, including malformed base64 or invalid JSON, so the test received the expected error and passed even after the HMAC comparison had been removed from the implementation.

The same project contained budget and targeting logic where CapAvailable always returned true and candidate selection was hardcoded to succeed. The methods TrySpend and the actual filtering code were never invoked on the request path; they existed only inside their own unit tests. These defects surfaced only during end-to-end runs.

Acceptance reports submitted by the agents occasionally contained commands such as “go test ./... ; echo EXIT:$?” or checks appended with “|| true”. Because the shell exit status came from echo, every report showed success. The reviewer now filters reports to discard any command that is not the final step or that suppresses non-zero exit codes.

Mutation testing also proved unreliable. In one case a sed mutation did not match due to escaping, leaving the original file unchanged; the “mutant killed” result was recorded anyway until file hashes were compared. Another mutation inverted a comparison scalar < 0.5 to scalar > 0.5, yet a loose assertion still passed because the new value remained below the threshold. A WordPress plugin survived a mutation that removed a return statement only because the PHPUnit bootstrap never loaded the plugin file under test.

Browser-based services passed all 31 mutations on the development machine yet failed every matrix cell on the staging environment because the Debian-packaged Chromium could not start its sandbox under the unprivileged user without CAP_SYS_ADMIN. Additional defects, including incorrect ClickHouse AggregatingMergeTree schema for UInt64 counters and value-versus-pointer passing of AppendStruct to clickhouse-go, also appeared only on the staging stand.

The current acceptance process therefore combines automated filtering of report commands, full diff review, manual execution on a staging stand, and cross-agent mutation testing where one model mutates code written by the other.