Google, Anthropic and Others to Support New AI Crawling Opt-Out Standard
The shared token will let publishers set one robots.txt preference for multiple AI companies as scraping pressure grows.
6 Articles
6 Articles
The biggest names in AI are backing a new way to opt out of scraping
OpenAI, Google, Microsoft, and Anthropic say they will adopt a new standard that publishers can use to opt out of being scraped by AI, as reported earlier by Search Engine Roundtable. With this control, publishers can apply their AI training preferences across multiple AI companies, instead of setting them individually. The shared "policy token" signal […]
The AI giants promised the media a unique way to prevent their content from being sucked.
Google, OpenAI, and Microsoft have agreed on a new robots.txt standard that restricts AI data collection. This makes content moderation easier for webmasters. The article "Google and OpenAI Set New Standard for AI Crawling" first appeared on TechInside.
Google, Anthropic and others to support new AI crawling opt-out standard
Google LLC, Microsoft Corp., Anthropic PBC and OpenAI Group PBC today announced plans to support an upcoming website crawling standard. The technology will enable website operators to block artificial intelligence developers from using their content to train their models. Many websites contain a text file titled “robots.txt” that informs crawlers what content they may access. […]
The proposed new solution will allow publishers to introduce a single standardized instruction that will be recognized and respected by AI companies that adopt the protocol.
Coverage Details
Bias Distribution
- 67% of the sources lean Left
Factuality
To view factuality data please Upgrade to Premium
