Server Log Analysis Reveals How Major AI Crawlers Actually Behave on Websites
A detailed review of server access logs shows that most AI vendors operate several distinct bots rather than a single crawler. Each bot serves a different function, and treating them identically in configuration files leads to unintended consequences.
GPTBot performs scheduled crawls for model training, respects robots.txt, and systematically follows sitemaps. OAI-SearchBot builds the search index used for answers that include citations. ChatGPT-User makes on-demand requests triggered by an individual user query. The same separation exists at other providers: Perplexity runs PerplexityBot for indexing and a separate user agent for live queries, while Anthropic maintains ClaudeBot alongside its own user-triggered agents.
Log analysis reveals three consistent request patterns. Index bots traverse site maps and internal links at regular intervals without correlation to traffic spikes. User agents issue single, targeted requests for specific URLs, often carrying a referrer from the AI service. All bots request robots.txt regularly, request sitemap.xml less frequently, and almost never request llms.txt. Over 515 million recorded AI-bot events, only 408 requests targeted llms.txt.
User-Agent strings are easily forged, so verification relies on reverse DNS or, preferably, comparison against IP ranges published by each vendor. A simple script that extracts IPs from logs and checks them against known OpenAI ranges quickly identifies spoofed traffic from unrelated scrapers.
Because llms.txt receives negligible attention from actual crawlers, site owners achieve better results by ensuring content remains in static HTML, pages load quickly, and robots.txt selectively permits user-agent traffic while blocking training bots. Structured headings and self-contained sections also improve the quality of excerpts returned to users.
Related articles
Nearly 70% of Companies Lack Control Over AI Agent Privileges, PAM Survey Finds
A recent survey conducted by AM Media among viewers and participants of the webinar 'How to Choose the Best PAM in 2026' shows that 68% of organizations fail to fully control privileges assigned to AI agents and automated services. Another 67% report insufficient oversight of API keys, tokens, and secrets, while 63% cite gaps in managing CI/CD and DevOps tool access. Traditional human administrator accounts are better protected at 34%, highlighting a shift in risk toward machine identities. Experts from NGR Softlab and Solar Group emphasize that service accounts often interact with multiple systems and require broad rights, making their compromise far more damaging than individual user breaches. JumpServer PAM CEO Denis Morozov noted that AI integration into PAM platforms could help map complex permission matrices and quantify risks for security teams. Participants also prioritized usability (72%) and ease of deployment (63%) when selecting PAM solutions over cost considerations.
Aligning AI Adoption Maturity with AI Security Using CMMI and Russian Regulatory Requirements
Organizations frequently operate at mismatched maturity levels between AI implementation and AI security controls, creating significant gaps that lead to incidents and regulatory violations. The analysis maps both dimensions onto the five-level CMMI scale, showing how adoption often outpaces protection and highlighting the risks of Shadow AI, uncontrolled agents, and non-compliance. Key frameworks including OWASP AIMA, Google SAIF, NIST AI RMF, and MITRE ATLAS are aligned to CMMI levels L1 through L5. Russian requirements under FSTEC Order No. 117, effective March 2026, introduce mandatory controls for trusted AI technologies, data handling, and statistical response validation that activate precisely at the L2-to-L3 transition. The article details predictable organizational states from denial to managed agentic systems and provides a practical matrix linking each CMMI level to minimum security requirements and specific Russian compliance anchors such as GOST R 56939-2024 and ISO/IEC 42001.
Study Finds Iterative AI Code Generation Accumulates Security Vulnerabilities Over Multiple Iterations
A 2025 IEEE-ISTAS 2025 research paper titled Security Degradation in Iterative AI Code Generation: A Systematic Analysis of the Paradox examined how repeated prompting of large language models leads to worsening code security. Researchers started with 10 secure code samples in C and Java, then applied four prompting strategies across 10 iterations each, generating 400 code samples that were analyzed with both manual review and automated scanners. The study found the strongest correlation between rising code complexity and vulnerability count, with 158 vulnerabilities emerging from feature-addition prompts and only 38 from explicit security-improvement requests. Even when asked to fix issues, GPT-4o frequently introduced new, subtler flaws such as timing side-channels, SQL injection risks, and use-after-free errors while addressing obvious problems. The authors recommend mandatory human review after every few iterations and greater use of SAST tools, noting that the illusion of progress can mask accumulating weaknesses. Limitations include testing only GPT-4o and the absence of human corrections during the iterative process.
Cybercriminals Deploy Advanced AI for Continuous Automated Reconnaissance and Exploitation at Scale
Advanced AI models now enable cybercriminal groups to maintain uninterrupted reconnaissance across enterprises in every sector, mapping domains, exposed services, and infrastructure changes at a pace no human team could sustain for weeks. The automated process targets two opposite profiles of vulnerable systems: legacy environments left unpatched for years with outdated versions and forgotten permissions, and rapidly deployed applications built through Vibe Coding that reach production without security review. The entire attack chain—reconnaissance, vulnerability identification, validation, and exploitation—is now executed by AI agents operating with minimal human oversight. What previously required a dedicated specialized team focused on one target at a time now runs in parallel against thousands of targets simultaneously, with marginal cost approaching zero for each additional attack. This collapse in operational costs and rise in success rates has restructured the cybercrime economy, funding increasingly sophisticated tools and lowering the barrier for new operators who need only platform access rather than deep technical expertise. The time window between an application reaching production and discovery by attackers has shrunk to hours, while most organizations still treat security as a finite project rather than an ongoing process.