AI cybersecurity is a cat and mouse gamejust now Russia to require content made by, with AI to be labeled as such — GFCN president1h ago Flipkart’s Super.money Bets on AI Agents to Outdo Bigger Rivals2h ago Datasette 1.0a39 and 0.65.4 security releases2h ago Anthropic caught scientists using Claude to further biological weapon research3h ago Oracle says AI will save it from the SaaSpocalypse, not bring it on3h ago Congress Is Suddenly Waking Up to the AI Doomsday Threat3h ago Don’t ask ChatGPT 'What should I do?' — this simple prompt helps you make better decisions yourself3h ago Extreme Networks’ Agent ONE Coworker moves AI networking from dashboards to answers3h ago Introducing ChatGPT for Financial Services3h ago Tesla unveils driverless Cybercab in Japan for 1st time4h ago The AI safety vibe shift4h ago Sam Altman met with top power utilities about securing the electrical grid. He offered one possible solution: OpenAI's cyber services.4h ago OpenAI Is Open to Slowing Cutting-Edge AI, CEO Sam Altman Tells Staff5h ago

AI cybersecurity is a cat and mouse gamejust now Russia to require content made by, with AI to be labeled as such — GFCN president1h ago Flipkart’s Super.money Bets on AI Agents to Outdo Bigger Rivals2h ago Datasette 1.0a39 and 0.65.4 security releases2h ago Anthropic caught scientists using Claude to further biological weapon research3h ago Oracle says AI will save it from the SaaSpocalypse, not bring it on3h ago Congress Is Suddenly Waking Up to the AI Doomsday Threat3h ago Don’t ask ChatGPT 'What should I do?' — this simple prompt helps you make better decisions yourself3h ago Extreme Networks’ Agent ONE Coworker moves AI networking from dashboards to answers3h ago Introducing ChatGPT for Financial Services3h ago Tesla unveils driverless Cybercab in Japan for 1st time4h ago The AI safety vibe shift4h ago Sam Altman met with top power utilities about securing the electrical grid. He offered one possible solution: OpenAI's cyber services.4h ago OpenAI Is Open to Slowing Cutting-Edge AI, CEO Sam Altman Tells Staff5h ago

AI Research8 min reading time

Only 8.9% of sites block AI crawlers, but 94.8% are never cited in AI answers

Hacker NewsPublished Aug 2, 2026

A study found that only 8.9% of websites block AI crawlers via robots.txt, yet 94.8% of audited sites never appear in AI assistant answers. Among assistants, Gemini cited audited businesses most frequently at 2.9%.

#ai-assistants #ai-crawlers #gemini #gptbot #oai-searchbot #robots-txt

More in AI Research

AI Research4 min read

An Anthropic researcher just quit, saying OpenAI and Anthropic are 'gambling with our lives'

Covered by 13 sources2d ago

AI Research10 min read

Anthropic Details Disrupted Claude Misuse Across Seven Harm Areas

Covered by 4 sources11h ago

AI Research4 min read

OpenAI asks Congress for mandatory national AI safety rules before it adjourns

Covered by 2 sources22h ago