Prompt Injection Emerges as Top Risk for LLM Applications in Production
Prompt injection attacks are shifting from lab demonstrations to production incidents that compromise AI assistants handling corporate data and code. The vulnerability arises because large language models receive all instructions in plain human language without reliable separation between developer rules and external content.
Why Models Confuse Commands and Data
Applications typically combine a hidden system prompt with user messages and retrieved documents before sending everything to the model. The model cannot distinguish trusted instructions from untrusted text such as emails, PDFs, or web pages. Classic examples show users appending phrases like “ignore previous instructions” to force unintended outputs. Indirect attacks hide instructions inside shared documents or repository comments, activating when the model summarizes or processes the file.
Real-World Incidents in 2025-2026
Microsoft 365 Copilot was shown leaking corporate data through a single email without user clicks. GitHub Copilot allowed a chain from a malicious comment in an external repository to code execution on a developer machine. Cursor faced issues with MCP configuration files where a poisoned document could insert backdoors. Earlier cases include a Chevrolet chatbot agreeing to sell a vehicle for one dollar and Air Canada losing a court case over chatbot responses. Slack AI demonstrated extraction of data from private channels via indirect injection.
Attack Vectors Without User Interaction
Attackers place hidden instructions in resumes, contracts, HTML comments, or image metadata. When a user requests a summary, the model executes the embedded command. Data can leak through URLs or images rendered in responses, sending secrets to attacker-controlled domains without any click. Substitutions in shared knowledge bases or tool-calling functions allow modification of external actions such as API calls or file changes.
Practical Defenses Recommended by Vendors
- Enforce permissions in code rather than prompts and grant models only minimum required access.
- Filter both inputs from external sources and outputs containing suspicious links or images.
- Require human confirmation for irreversible actions such as payments, deletions, or external publications.
- Explicitly mark external content in prompts to reduce confusion between developer instructions and retrieved data.
- Test against malicious PDFs, emails, and repository comments, not only direct chat attempts.
- Limit automatic image loading and restrict the model’s ability to publish or modify data without oversight.
Meta advises against combining untrusted external reading, access to sensitive data, and external modification in a single session. NIST research confirms that any finite set of static rules remains bypassable by adaptive attackers. OpenAI, Anthropic, and Google report that most published defenses were defeated in real adaptive scenarios with success rates exceeding 90 percent.
Organizations are advised to treat AI agents like web applications accepting untrusted input and to maintain continuous testing and layered controls rather than relying on a single patch.
Related articles
Microsoft Copilot Abused to Accelerate Corporate Account Takeovers and Bank Transfer Frauds
Cybercriminals can leverage Microsoft Copilot to speed up the compromise of corporate accounts and execute financial frauds. In a controlled simulation, researchers demonstrated how the AI assistant helped redirect a $247,500 bank transfer after gaining access to an employee's email. The attack begins with the creation of inbox rules that automatically delete suspicious login alerts, followed by rapid analysis of internal communications to identify high-value targets. Copilot was then used to replicate the victim's writing style and craft a convincing phishing email capable of stealing session tokens and bypassing multi-factor authentication. Once the executive account was taken over, the tool quickly located pending payments and generated a fraudulent request to alter the recipient bank details. The scenario highlights risks from legitimate AI features rather than any specific vulnerability in Copilot itself.
Vibe Hacking Rises: Generative AI Lowers Barriers for Offensive Security Operations
Generative AI is reshaping the cybersecurity threat landscape by dramatically reducing the cost of offensive security knowledge. Previously, attackers needed years of experience in reverse engineering and exploit development to conduct meaningful intrusions, but AI now assists with research, code generation, debugging, and adapting known techniques to new environments. This enables less experienced individuals to build functional attack chains in weeks rather than years. The article introduces the concept of vibe hacking, where attackers collaborate with AI assistants using natural language prompts in a process similar to vibe coding. Defenders can no longer rely on the scarcity of skilled adversaries and must shift toward continuous validation frameworks such as CTEM, AEV, and PTaaS. Senior human experts remain essential for risk judgment and business context, even as automation accelerates routine tasks.
Hunt.io Exposes Suspected Chinese Cyber Espionage Operation Using Agentic LLMs Claude Code and DeepSeek
In July 2026 Hunt.io published research on a suspected Chinese cyber espionage campaign uncovered through an exposed directory on a Hong Kong server. The leak contained 2,431 files including victim source code, operation logs, web shells, exploitation scripts, scan results and phishing page clones. Researchers identified traces of Claude Code and DeepSeek-v4-pro working together, with Claude Code handling agentic tasks and session context while DeepSeek supported reasoning, script refinement and next-step selection. The infrastructure cluster, known as TencShell, showed overlapping SHA-256 HTTP headers, SSH host keys and TLS certificates across 13 IP addresses. Multiple initial access vectors were observed, including SQL injection against Taiwanese and Thai government targets plus exposure of Supabase and Azure secrets. The report also references a parallel Anthropic disclosure on GTG-1002, another Chinese state-linked operation that used Claude Code for 80-90 percent of tactical work.
OWASP LLM10 Unbounded Consumption: Testing YandexGPT Lite 5 and GigaChat Lite on Resource-Intensive Prompts
The article examines OWASP LLM10: Unbounded Consumption, a category focused on attacks that force large language models to perform computationally expensive tasks without traditional denial-of-service traffic. Researchers used the AI Red Team platform to evaluate two flagship Russian models, YandexGPT Lite 5 and GigaChat Lite, against prompts designed to trigger recursive context expansion, fractal nesting, combinatorial explosions, and multi-agent simulations. The study highlights how a single carefully crafted prompt can generate disproportionate token consumption, GPU load, and API costs compared with conventional DoS methods. Both models demonstrated defensive strategies including hard generation limits at 512 tokens, graceful degradation by summarizing instead of expanding, and early refusal with alternative suggestions. The work was supported by the Foundation for Assistance to Innovations and aims to improve architectural protections for LLM applications. Examples of tested prompts and observed model behaviors are provided for research and educational purposes only.