Back to Home
The Verge AI··Industry Media

The biggest names in AI are backing a new way to opt out of scraping

中文摘要

OpenAI、Google、微软和 Anthropic 将采用新标准,让出版商能通过“策略令牌”一次性拒绝多家 AI 公司的抓取。

English Summary

OpenAI, Google, Microsoft, and Anthropic are adopting a new standard, letting publishers opt out of AI scraping across multiple companies simultaneously via "policy tokens."

Original Excerpt

OpenAI, Google, Microsoft, and Anthropic say they will adopt a new standard that publishers can use to opt out of being scraped by AI, as reported earlier by Search Engine Roundtable. With this control, publishers can apply their AI training preferences across multiple AI companies, instead of setting them individually. The shared "policy token" signal is proposed by Kevin Kelley, a technical lead for standards at Anthropic, as part of the Internet Engineering Task Force (IETF). It will live within a website's robots.txt file, which has long signaled which web crawlers can and can't scrape a site - whether for search indexing, archiving, or … Read the full story at The Verge.