AI Crawlers Devour Web Traffic as Scraping Ratios Hit 38,000 Pages per Human Visitor
AI-powered crawlers have transformed web traffic patterns, with robots now accounting for the vast majority of requests on many sites. One developer tracking charitable donors on PatronView measured 1.28 million page deliveries where fewer than 6,000 came from actual humans, producing a ratio of roughly 214 automated loads per human view.
Four main categories of bots are active. Honest SEO crawlers such as SemrushBot and AhrefsBot identify themselves and index content normally. Training bots like GPTBot and ClaudeBot perform bulk downloads to feed large language models. Search-oriented AI crawlers (Claude-SearchBot, Amzn-SearchBot) and agentic bots executing live user prompts generate the heaviest load.
In April, PatronView recorded 3.6 million requests in a single day from 361,844 unique IPs concentrated in China. Cloudflare blocked 1.18 million attempts over ten hours, yet a troubling share still bypassed CAPTCHA challenges. The site owner ultimately blocked entire countries at the edge, later adding Vietnam and Singapore, because 95.9 percent of its real audience is in the United States.
Traffic exchange ratios have collapsed. Cloudflare previously reported Anthropic crawlers at 3,000 pages per visitor; measurements on PatronView reached 35,000 to one. Claude-SearchBot fetched 420,680 pages in a week while Claude-User delivered only 12 humans. Amazon’s Rufus-related crawler requested 117,000 pages daily and returned zero visitors.
Founder Bruce Nash of The Numbers observed even steeper imbalances: Google at roughly five pages per visitor, OpenAI above 1,000, and Anthropic exceeding 38,000. The 1997-era film database suffered a week-long outage in March 2026 after scraping campaigns escalated into targeted reconnaissance for Polymarket prediction markets. The team ultimately rebuilt a reduced version on new infrastructure rather than risk repeated compromise of legacy code.
Related articles
Raft Develops Multilabel Guardrail Classifier Detecting 15 Risk Categories with 3x Speed and Cost Gains
Raft has released details on a custom multilabel guardrail classifier designed to scan both incoming prompts and model outputs for 15 distinct risk categories in real time. The system handles Russian and English text while maintaining independent thresholds for each category to balance false positives against critical misses. By switching from a PyTorch baseline to TensorRT inference on NVIDIA RTX 3090 hardware, the team achieved a 2.95x reduction in single-request latency and lowered inference cost to $0.062 per million requests. The architecture uses per-category expert query tokens plus a lightweight interaction transformer to capture correlations such as those between armament and violent content. Training relied on asymmetric loss functions and post-epoch per-class threshold tuning rather than standard binary cross-entropy. Benchmarking against nine open guardrail and toxicity models showed superior macro-F1 on rare but high-impact categories while remaining an order of magnitude cheaper than external LLM judges.
Zhou Hongyi Warns AI Tools Are Industrializing Vulnerability Discovery
At the Fourth Cyberspace Security Forum in Tianjin, 360 founder Zhou Hongyi stated that vulnerability mining is shifting from artisanal workshops to automated production lines, compressing discovery cycles from months or years down to hours. AI tools such as Mythos are standardizing and automating the process, enabling attackers to replicate elite hacker expertise at scale through distilled models and agent swarms. 360's own Tulongfeng platform has already discovered over 10,000 vulnerabilities since its June release, including long-hidden high-risk flaws in Windows, Office, OpenClaw, Flowise, and Codex. The emergence of multi-agent systems introduces new attack surfaces because compromised agents can autonomously collaborate and move laterally faster than human operators. Zhou described this as the "second one-way transparency," where offensive tradecraft becomes copy-pasteable via prompts and toolchains. Defenders are advised to adopt "model-versus-model" strategies, automate vulnerability intelligence workflows with SOAR, enforce strict agent permission audits, and integrate AI into their own code review and detection engineering processes.
Anthropic Fable 5.1 System Prompt Fully Leaked Hours After Launch Exposing 275000 Characters of Rules
Anthropic released its flagship Fable 5.1 model alongside Mythos 5.1 on September 2, achieving strong benchmark scores including 90 percent on ARC-AGI-2. Within hours, researcher Pliny the Liberator published the complete 275000-character system prompt on GitHub, far exceeding the company's official 27000-word disclosure. The leaked document details 46 built-in tools, strict copyright restrictions, memory classification boundaries, and behavioral constraints that function as an internal employee handbook. The incident highlights that model weights remain the true core while prompt-based guardrails create an attack surface once mapped. It also reveals privacy rules that permanently exclude storage of minor identities, criminal records, and self-harm indicators even when users disclose them. The leak underscores the growing gap between vendor transparency claims and actual runtime instructions governing frontier AI systems.
Fine-Tuning Prompt Injection Detector: Five Rounds, Four Failures and Gates Against Regression
AGIMA technical director Andrey Nepryakhin details the challenges of adapting an open-source GLiNER-based prompt injection detector for Russian corporate traffic. The system processes nearly one million requests monthly from eighty employees using agentic tools. Five fine-tuning rounds using LoRA adapters on microsoft/mdeberta-v3-base were conducted, yet only the second round reached production. Failures stemmed from regression on previously fixed attacks, imbalanced negative samples, and reliance on rule-based signals instead of the model. The team introduced cumulative corpora, person-based splits, and ablation checks against the base model to prevent silent rollbacks. A gradient mass rule limits any single phenomenon to 10-20 percent of training data to avoid precision or recall collapse.