Study Finds Iterative AI Code Generation Accumulates Security Vulnerabilities Over Multiple Iterations
A new study presented at IEEE-ISTAS 2025 demonstrates that iterative prompting of large language models for code generation causes progressive degradation of security properties.
Researchers began with 10 secure, vulnerability-free code samples written in C and Java. They then applied four distinct prompting strategies—adding features, optimizing performance, explicitly improving security, and using vague “make it better” requests—across 10 successive iterations, producing 400 code variants that were examined both manually and with automated scanners.
Key Findings on Vulnerability Accumulation
The number of vulnerabilities increased nonlinearly with each iteration. Prompts focused on adding functionality generated the highest total (158 vulnerabilities), while security-oriented prompts produced the lowest count (38). Nevertheless, even security-focused requests often replaced obvious flaws with more subtle ones.
A statistically significant positive correlation (r = 0.64) was observed between code complexity growth and vulnerability count. For every 10 % increase in complexity, researchers recorded an average 14.3 % rise in vulnerabilities.
Concrete Examples of Degradation
- Memory-management functions evolved from safe allocation routines into complex pointer arithmetic containing buffer-overflow and use-after-free risks by iteration 10.
- Authentication token checks acquired timing side-channels, parsing vulnerabilities, and flawed multi-factor recovery logic.
- Database access routines lost parameterization, introduced dynamic string concatenation, and later added transaction support that created race conditions.
Although 27 % of early security-oriented iterations (iterations 1–3) produced genuine improvements such as added input validation and NULL checks, these gains were typically offset by new hidden vulnerabilities in later rounds.
Recommendations and Limitations
The authors conclude that human oversight after every iteration—or at minimum after no more than three iterations—is essential. They also advise increased reliance on static application security testing (SAST) tools whenever code volume grows substantially.
The study tested only GPT-4o and did not allow human corrections during the iterative process, leaving open questions about whether newer models exhibit the same pattern at later iteration counts.
Related articles
OpenAI GPT-6 Astra Deploys Multi-Agent Parallel Processing, Increasing Local CPU Load and Security Risks
Early users of GPT-6 Astra have observed the model distributing complex tasks across multiple specialized agents that plan, solve, test code, verify results, and iterate after failures. This multi-agent approach enables faster handling of multi-step workflows compared to sequential chatbots. OpenAI states that Astra can control computers, operate browsers and applications, and install or test software, though it has not officially confirmed a native multi-agent architecture. Main computations run in the cloud, but agent tools can execute on user devices or corporate servers, leading to noticeable processor load when multiple agents compile code, launch browsers, run tests, and operate containers simultaneously. Corporate environments face added complexity as each agent requires virtual machines, sandboxes, internal data access, and careful environment cleanup. The increased autonomy has prompted OpenAI to strengthen monitoring of Astra actions and permission boundaries for subscribers of ChatGPT and enterprise clients.
Microsoft Copilot Can Surface Overshared Data Despite Permission Boundaries
Microsoft documentation states that Copilot only accesses data authorized for the signed-in user, yet default SharePoint and OneDrive sharing settings often grant broad access that the AI then respects literally. This creates accidental oversharing risks where Copilot retrieves documents shared too widely years earlier. Administrators can use Content Management Assessment and Data access governance reports, including the EEEU report covering the top 100 sites shared in the past 28 days, to identify problematic content. Two distinct controls exist: Restricted Access Control removes access entirely while Restricted Content Discovery hides items from Copilot and search without altering permissions. Sensitivity labels combined with encryption can exclude programmatic access for agents, though Microsoft does not guarantee outright blocking. Interaction logs stored in Microsoft Purview retain user prompts, Copilot responses, and citations to accessed documents, providing an audit trail for oversharing incidents.
Adaptive LLM Worm Uses Local Models to Craft Per-Target Exploits in Heterogeneous Networks
Researchers from the University of Toronto have published a preprint describing an adaptive computer worm driven by LLM agents that spreads across corporate networks by generating individualized attack strategies for each compromised system. Unlike traditional worms such as WannaCry that rely on fixed exploits, this worm maintains its own infrastructure by running local LLMs on infected GPU-equipped machines to analyze vulnerabilities and synthesize new attack vectors in real time. The system was tested in an isolated FakeCorp environment containing Linux, Windows, and IoT devices, successfully leveraging known real-world vulnerabilities to propagate over 48 hours and seven-day autonomous runs. Two core components power the worm: a GPU-hosted LLM component and a hierarchical agent framework with memory, reasoning graph, and tool modules that manage reconnaissance, exploitation, and payload deployment. The authors note that the approach creates an economic asymmetry favoring attackers because the worm parasitizes victim compute resources, eliminating the need for external C2 or commercial LLM services. They warn that adding adaptive reasoning to historical worms such as SQL Slammer, Conficker, or Stuxnet would significantly increase their resilience while remaining slower and noisier than classic self-propagating malware.
Building Secure On-Prem AI Assistants: How to Keep Corporate Data Inside Closed Contours
Many organizations hesitate to deploy AI assistants due to strict data protection rules that prohibit sending information to external clouds. The article explains how to implement AI models entirely within a company's own infrastructure, ranging from on-premise servers to fully offline laptops. It breaks down four deployment locations from public APIs to local devices and clarifies three distinct access levels: read, write, and execute. The author emphasizes that most business value comes from read-only access combined with human-in-the-loop controls for any irreversible actions. Practical recommendations include RAG over model size, quantization for local hardware, and maintaining immutable audit logs. The piece also warns that preparing clean knowledge bases often consumes more effort than the model itself.