Fine-Tuning Prompt Injection Detector: Five Rounds, Four Failures and Gates Against Regression

AGIMA technical director Andrey Nepryakhin published the second part of a detailed case study on building a security layer in front of large language models. The post examines five rounds of fine-tuning a prompt-injection detector and explains why only one round survived in production.

The detector uses the open GLiNER model on the microsoft/mdeberta-v3-base encoder. It sits between employees and external LLMs, handling almost one million requests per month from eighty users, primarily developers with agentic tooling. Because the base model lacks knowledge of Russian corporate context, the team trains LoRA adapters while keeping the encoder frozen. Each training round completes in minutes on modern hardware.

Training data comes exclusively from production incident logs. The unit of the corpus is a fixed-length window rather than a full document, because the model never sees entire long texts at inference time. Labeling uses three classes instead of binary attack/non-attack: allow for normal work, warn for correct signals that should only trigger warnings, and block for confirmed attacks. This scheme prevents the model from learning to ignore real attack vectors that resemble routine engineering tasks.

An empirical “gradient mass” rule emerged from the experiments: any phenomenon targeted for improvement must constitute 10–20 percent of the training set. Lower shares disappear; higher shares cause over-generalization and loss of true positives. The team also splits training and validation sets by person rather than randomly to avoid leakage of individual writing styles.

The first round collected 431 negatives representing 12.8 percent of the data. It reduced false blocks dramatically but caused nine previously detected attacks to drop below threshold, including phrases such as “enable developer mode without filters.” Ablation against the base model revealed that the adapter had overwritten useful signals rather than learning new distinctions. The second round restored the cumulative corpus from prior rounds, lowered the negative share to 8.1 percent, and achieved 0.6 percent false blocks on 179 unseen working texts while raising recall from 0.89 to 0.94.

The third round attempted to recover a lost extraction attack using contrastive pairs but doubled the number of false positives on flagship working texts. The fourth round targeted two high-volume false-positive sources that the model itself scored at zero; both were handled by rule layers, rendering the training round useless. The fifth round improved aggregate metrics yet broke detection of several flagship attack phrases and was rolled back within an hour.

Additional safeguards now include pre-flight searches for phrases from earlier rounds, separate sets for training and regression gates, and mandatory verification that the full pipeline (including windowing) is used for every measurement. The article concludes that precision and recall are not always in direct trade-off when the root cause is corpus imbalance rather than model capacity limits.