Can Open-Source AI Agents Replace Pentesters? Strix and PentAGI Tested on Real 200-Host Perimeter

An experienced bug hunter tested whether open-source LLM agents can replace human pentesters by running Strix and PentAGI against a real customer external perimeter of approximately 200 active hosts. The evaluation used the GLM 5.3 model and focused on whether the agents could perform reconnaissance, vulnerability discovery, proof generation, and reporting with minimal human intervention.

Both agents operate inside isolated Docker sandboxes, plan their own tasks, launch tools, create PoCs, and compile reports. Strix is optimized for web-application and code analysis, while PentAGI supports longer contexts and more comprehensive perimeter testing. The researcher deliberately avoided benchmarks and instead used live infrastructure to measure practical performance.

Reconnaissance Limitations

Neither tool performs high-level reconnaissance from abstract inputs such as organization names or IP ranges. PentAGI was given a broad prompt and spent 6 hours 41 minutes normalizing the scope, resolving 177 hostnames, scanning two CIDR blocks, fingerprinting services, and enumerating 280 web applications. It correctly identified a transparent middlebox that invalidated SYN scans and switched to application-layer probes, yet it continued scanning a foreign subnet after detecting a scope-boundary violation, requiring manual termination.

Vulnerability Findings and False Positives

On identical scope, Strix ran for 3.8 hours and produced 15 findings. After manual review only one was rated Medium, two Low, and twelve Informational; the single High finding was public market statistics that the organization already discloses. PentAGI ran for 9.2 hours and reported 296 vulnerabilities, including 42 Critical and 50 High. None of the 92 high-severity items were actually exploitable; several “confirmed-with-PoC” entries referenced hardcoded template hashes that never existed in executed proofs.

Real critical issues existed on the perimeter, including an RCE in a document converter and an OAuth secret disclosure that enabled code execution chains. These were found only by a commercial autonomous vulnerability scanner.

Context Management and Token Costs

Strix consumed 145.1 million tokens because it spawned 41 sub-agents, each receiving repeated system prompts and rules of engagement. PentAGI used aggressive summarization (279 summarizer calls) and consumed 31.4 million tokens, but the compression introduced detail loss and looping risk on larger scopes.

The researcher concluded that both agents can assist with targeted checks yet cannot replace human control of scope, triage of false positives, or accurate risk assessment. Architectural issues around memory growth in Strix and context degradation in PentAGI remain unsolved for production use.