The biggest names in AI are backing a new way to opt out of scraping

OpenAI, Google, Microsoft, and Anthropic will adopt a new shared 'policy token' standard allowing publishers to opt out of AI training data scraping via a robots.txt file signal. This unified approach simplifies control over AI crawler access across multiple companies, with Microsoft offering granular options for AI training and grounding. The standard is proposed by Anthropic's Kevin Kelley and will roll out starting next year.