Fine-Tuning Prompt Injection Detector: Five Rounds, Four Failures and Gates Against Regression
AGIMA technical director Andrey Nepryakhin published the second part of a detailed case study on building a security layer in front of large language models. The post examines five rounds of fine-tuning a prompt-injection detector and explains why only one round survived in production.
The detector uses the open GLiNER model on the microsoft/mdeberta-v3-base encoder. It sits between employees and external LLMs, handling almost one million requests per month from eighty users, primarily developers with agentic tooling. Because the base model lacks knowledge of Russian corporate context, the team trains LoRA adapters while keeping the encoder frozen. Each training round completes in minutes on modern hardware.
Training data comes exclusively from production incident logs. The unit of the corpus is a fixed-length window rather than a full document, because the model never sees entire long texts at inference time. Labeling uses three classes instead of binary attack/non-attack: allow for normal work, warn for correct signals that should only trigger warnings, and block for confirmed attacks. This scheme prevents the model from learning to ignore real attack vectors that resemble routine engineering tasks.
An empirical “gradient mass” rule emerged from the experiments: any phenomenon targeted for improvement must constitute 10–20 percent of the training set. Lower shares disappear; higher shares cause over-generalization and loss of true positives. The team also splits training and validation sets by person rather than randomly to avoid leakage of individual writing styles.
The first round collected 431 negatives representing 12.8 percent of the data. It reduced false blocks dramatically but caused nine previously detected attacks to drop below threshold, including phrases such as “enable developer mode without filters.” Ablation against the base model revealed that the adapter had overwritten useful signals rather than learning new distinctions. The second round restored the cumulative corpus from prior rounds, lowered the negative share to 8.1 percent, and achieved 0.6 percent false blocks on 179 unseen working texts while raising recall from 0.89 to 0.94.
The third round attempted to recover a lost extraction attack using contrastive pairs but doubled the number of false positives on flagship working texts. The fourth round targeted two high-volume false-positive sources that the model itself scored at zero; both were handled by rule layers, rendering the training round useless. The fifth round improved aggregate metrics yet broke detection of several flagship attack phrases and was rolled back within an hour.
Additional safeguards now include pre-flight searches for phrases from earlier rounds, separate sets for training and regression gates, and mandatory verification that the full pipeline (including windowing) is used for every measurement. The article concludes that precision and recall are not always in direct trade-off when the root cause is corpus imbalance rather than model capacity limits.
Related articles
Docker Rebrands Around AI Agents with 'Run on Trust' and New Sandbox Tools
Docker has updated its branding to emphasize security for AI agents, introducing the slogan 'Trust Docker for the agents you don’t' and the tagline 'build, ship, and run on trust'. The company highlighted products such as Sandboxes, AI Governance, MCP Enterprise Gateway, and Hardened Images, although most of these features were released or discussed earlier in 2025. Sandboxes run agents inside microVMs with separate kernels using KVM or Windows Hypervisor Platform, allowing isolated execution of tools like Claude Code. AI Governance provides centralized policy management and audit logging for enterprise teams, while MCP Enterprise Gateway handles SSO and credential injection for MCP servers. Hardened Images remain available under Apache 2.0 with optional paid FIPS and support tiers. The shift follows the appointment of Don Johnson, formerly of Oracle, as CEO and reflects Docker’s attempt to address new security concerns around autonomous AI agents rather than traditional container workloads.
AI Agents Exploit PaperCut Flaw, Compromising 395 Organizations Across 48 Countries
PaperCut released a security bulletin on 27 August 2026. GreyNoise observed the first real-world code execution just four days later, with the operator moving from initial access to execution in under four hours. After the mass campaign launched, 11 organizations were compromised in only 26 seconds. In total, at least 440 PaperCut NG/MF instances belonging to 395 organizations were breached. The incident is detailed in the November report by Google Threat Intelligence Group and Mandiant, which examines the role of AI agents in the attacks. GTIG also discovered an automated exploit_agent system capable of handling exploitation without constant human input. Additional cases involved LLMjacking and unauthorized configuration changes in Claude Code repositories at cryptocurrency firms.
Google Threat Intelligence Group Warns of Fully Autonomous AI Cyberattacks Within Six Months
Google's Threat Intelligence Group (GTIG) has observed the first real-world instances of attackers using multi-agent AI frameworks to autonomously discover vulnerabilities, generate exploits, and compromise targets without human intervention. The discovery came from an exposed server running a framework called exploit_agent that analyzed software update logs from popular web applications to identify and exploit weaknesses. Four previously defensive tools—ARTEX, CAIRN, CyberStrikeAI, and HexStrike AI—have been repurposed for offensive operations, with documented cases targeting French media, U.S. aerospace, and cryptocurrency firms. GTIG projects that low-complexity automated attacks will become common within three months, while sophisticated fully autonomous operations are expected within six months. The report highlights how these systems compress patch windows, overwhelm security teams with AI-generated proof-of-concepts, and turn detailed vendor advisories into attack playbooks. Defenders are urged to minimize public exposure, accelerate patching cycles to weekly, and tightly control API keys to prevent LLMjacking.
OpenAI Rolls Out KYC, Hardware Keys and Tiered Access for Cybersecurity Tasks in GPT-6.2
OpenAI has introduced stricter access controls for GPT-6.2 Astra and Sol models when users attempt cybersecurity-related tasks. Certain legitimate security research and testing activities now trigger automatic blocks unless the user obtains Daybreak trusted access. The program requires paid subscriptions, FIDO2 hardware keys, Advanced Account Security, and identity verification through Persona. Developers report that even after meeting preliminary requirements, access to reduced refusal rates is not guaranteed and may require separate organizational approval. The changes stem from the models reaching Critical capability levels under OpenAI’s Preparedness Framework for dual-use cybersecurity skills. Similar verification processes previously led to account blocks for users of competing services such as Anthropic’s Claude.