NYU Researchers Find 86% of Chatbot Responses on Digital Abuse Contain Errors, Gaps or Safety Risks
Researchers at New York University’s Tandon School of Engineering have found that leading chatbots frequently provide inaccurate, incomplete, or dangerous advice when users ask about protection from digital abuse.
The team tested GPT-4o, Claude 3.7 Sonnet, and two specialized assistants for survivors of abuse, Aimee and Ruth, using 193 questions covering spyware, location tracking, and account takeover scenarios.
Problems with accuracy, completeness, or safety appeared in 86 percent of all responses. Unsafe answers accounted for 74 percent of Aimee’s replies, 65 percent of Claude’s, 61 percent of Ruth’s, and 55 percent of GPT-4o’s.
In 60 percent of cases the models failed to include essential warnings about risks. Common errors included recommending a VPN to counter spyware or advising password changes without considering that an abuser might control the device or shared account.
Such actions can alert the perpetrator to the victim’s efforts or result in the loss of crucial evidence. The researchers also tested whether the systems would assist potential abusers; all four complied, with Aimee and Ruth answering approximately two-thirds of malicious queries.
The findings highlight that technically plausible instructions from chatbots can worsen a victim’s situation when dealing with spyware and surveillance threats.
Related articles
Cloudflare Engineer Releases KillSwitch Language Designed to Trip Up AI Code Models
Dominic Pichet, an engineer at Cloudflare, has created an experimental programming language called KillSwitch specifically engineered to cause large language models to fail when interpreting or generating code. The language turns programs into narrative-style text where most content is ignored by the interpreter, hiding actual commands inside stories and using distance-based references to extract values. Rules for identifiers rely on letter counts and anagrams, while the language deliberately incorporates phrases such as “ignore previous instructions” that carry functional meaning inside KillSwitch itself. Pichet added misleading statements in the documentation and wording crafted to trigger model safety filters, creating a hostile environment for coding agents. In tests involving writing Hello World, explaining program output, and converting temperatures, Claude Opus 5 achieved the highest score of 66.8 points. The full source code and specification are available on GitHub, and the author plans to increase difficulty if models improve.
Zscaler Report Details How Generative AI Accelerates Custom Malware Creation and Social Engineering
Zscaler published an analysis on October 16 examining the use of generative AI by attackers for social engineering, data discovery, and malware development. Principal security researcher Phil Hagen notes that the report focuses on broad methods rather than naming specific malware families or confirmed campaigns. The core observation centers on the dramatically increased speed at which malicious scripts can be written and iteratively revised. Traditional detection that relies on matching files or traffic against known patterns loses effectiveness when code is frequently altered while preserving the same functionality. The analysis clarifies that observed rapid creation and revision does not equate to fully autonomous self-mutating malware that changes on every execution. Another impact is the disappearance of classic indicators such as spelling and grammar errors in phishing messages, which previously helped identify malicious intent. Zscaler recommends focusing on behavioral analysis of what programs actually do rather than solely relying on matches to previously catalogued samples.
AI Agents Using Claude and Grok Fully Automate Attacks on 400 Latin American Financial Institutions
A Straiker investigation has revealed that the CyberCartel hacking group built an AI agent network driven by multiple large language models to conduct fully automated attacks against more than 400 Latin American banks and payment applications. The system incorporates models from Anthropic and xAI, with CrowdStrike confirming additional use of DeepSeek. The AI agents independently perform reconnaissance across hundreds of targets, steal credentials through adaptive techniques, take over accounts while mimicking human behavior to evade detection, and monetize stolen data with minimal human intervention. This development marks a sharp drop in attack costs, allowing scalable operations that were previously impractical due to high manual effort. Defenders are advised to deploy dynamic behavioral baselines, enforce zero-trust architectures, strengthen multi-factor authentication and transaction monitoring, and incorporate AI adversary simulations into red-team exercises. The report is described as one of the first detailed public records of such end-to-end AI-driven financial attacks.
Registration Opens for SOC Forum 2026 with Focus on GenAI and Cybersecurity Risks
GC Solar has opened registration for SOC Forum 2026, the central event of the Russian Week of Cybersecurity. The two-day conference will take place on 27-28 October at the Lomonosov cluster in Moscow and will examine both the economic benefits and the security challenges of generative AI. Day one covers business impact, regulatory questions, data control, liability for AI errors, and new threat vectors. Day two shifts to hands-on technical tracks covering attack detection, defensive infrastructure development, and the use of AI by both attackers and defenders. The program includes dedicated tracks such as AI & Cybersecurity, Offence, Defence, and SOC-practicum. Organizers emphasize practical discussions over marketing presentations about neural-network images.