Hermes Emerges as Modular Harness for Practical AI Security Testing

The language model by itself possesses no inherent security capabilities. The earlier this illusion disappears, the sooner a functional tool can appear. Running any frontier model out of the box and asking it to test an agentic system or craft a current attack vector produces instantly disappointing results. The model outputs naïve jailbreak attempts reminiscent of 2023, long after the industry moved past GPT-3.5-era thinking.

In real AI security work, base models remain anchored to stale weights, confuse hallucinations with actual vulnerabilities, and know nothing about recent long-term memory poisoning techniques, sandbox escapes, or fresh exploits in model repositories. The author, who runs the PWN AI Telegram channel, stresses that the field changes weekly. Without external scaffolding, the model stays a disconnected text generator. Only the combination of a language model and a specialised harness produces a dependable, reproducible agent.

After evaluating monolithic frameworks such as OpenClaw, the author rejected them because of excessive abstractions, token waste, and state leakage. The choice fell on Hermes for its modularity, dynamic skill loading, and minimal overhead. The harness runs on DeepSeek-v4-flash (April and July releases) chosen for strong agentic performance and low cost. A separate DeepSeek harness was tested but found too immature for production use.

The architecture rests on three control loops plus an execution layer. The management layer pins the agent to version-controlled runbooks, skills and plans stored in Git so every step remains auditable and reversible. A background data-collection layer continuously queries more than thirty sources—NVD, CISA KEV, Exploit-DB, Unit 42, Resecurity, arXiv and others—writing timestamped artefacts to disk. An analysis layer applies Hermes skills that clean noise, rank threats with the custom TIPS metric, and produce compact JSON summaries.

Execution occurs inside a restricted Docker sandbox connected to security tools including Garak, PyRIT, promptfoo, fickling, modelscan and presidio. Context inflation is fought through dynamic skill injection, report deduplication, and replacement of raw logs with structured JSON. Eight mandatory validation gates must all turn green before the model is allowed to emit a final answer.

Academic papers undergo a four-stage pipeline: candidate discovery via arXiv and Semantic Scholar, expert filtering, deep technical extraction by the Paper-to-PoC skill, and vision analysis of diagrams. Surviving findings are reproduced against a local test model under strict network and command restrictions. Successful reproductions are automatically committed back to the repository, allowing the harness to learn from its own results.