Tech

Google and Microsoft among those signing up to OpenAI's plan for a new website crawling standard — including opting-out of internet scraping

TechRadar By Rahim Amir Nov 8, 2026 · 11:05 AM 5 min read 0 views 0 comments

Google and Microsoft among those signing up to OpenAI's plan for a new website crawling standard — including opting-out of internet scraping — Image: TechRadar

One robots.txt line could soon tell Google, Microsoft, OpenAI, and Anthropic not to train on your site, but it won't stop their bots from crawling it.

TechRadar Original story · www.techradar.com

Read the full story

How does this story make you feel?

👍 ❤️ 💡 😮 😂 😢 😠

Discussion (0)

No comments yet. Start the conversation!

Discussion (0)

More to read

Google, Anthropic and others to support new AI crawling opt-out standard

Lifetime access to Microsoft Word, Excel, PowerPoint, and more is now just $49.97

Does opting out of TV tracking work?