Troubleshooting AI Bug Hunting: Seven Critical Mistakes That Destroy Report Validity

AI-assisted bug hunting has moved from novelty to standard practice on Russian bug bounty platforms. In a detailed OFFZONE talk later published as an article, researchers Vladislav Bek and Daniil Shushakov (t.me/LanyacBug) outlined seven recurring failures that turn promising AI output into rejected or low-value reports.

The first and most common issue is hallucination. Agents invent vulnerabilities, inflate CVSS scores to 10.0, and present unverified assumptions as facts. The recommended countermeasure is to grant agents real execution capability inside isolated virtual machines or carefully scoped MCP containers so they can run commands and observe actual results instead of guessing.

A second major problem occurs when hunters skip manual validation. Blind trust, time pressure, and low personal cost for invalid submissions lead researchers to submit unverified chains. The authors advise writing custom validation rules for each vulnerability class (injections, SSRF, IDOR, file uploads) and always reproducing findings with Burp or curl before submission.

Many reports also suffer from a missing threat model. Agents treat any information disclosure as critical and build speculative attack chains without business context. Solutions include supplying the agent with explicit application context and deploying a second agent whose sole task is to disprove the first agent’s claims.

Severity inflation is another frequent error. Agents default to worst-case scenarios and assign critical ratings without demonstrating real impact. The fix is to require agents to prove the claimed effect through concrete steps and to calculate CVSS vectors manually rather than accepting model-generated scores.

Reports often become bloated with copied CWE and OWASP text. The researchers impose a strict five-sentence limit per section and force the agent to describe the exact attacker actions and resulting business damage in plain language.

Cost management is addressed through cascaded models: expensive frontier models handle planning and analysis while cheaper models perform scanning and data collection. Valid findings and negative results are stored for reuse, reducing repeated token spend. HexStrike and custom Claude-based agents with Chromium extensions form the current practical stack.

Finally, agents frequently drift out of scope. Clear rules for wildcard, domain-list, and single-domain programs, combined with continuous human oversight, keep exploration inside program boundaries and preserve researcher reputation with vendors.