‘We believe this is a practical step toward an open web’: Don't want AI companies training on your website? OpenAI, Google and Microsoft are backing a new way to opt out
It just got easier to block AI web scraping
by https://www.techradar.com/author/benedict-collins · TechRadarFeatures By Benedict Collins Published 3 November 2026
Share this article 0 Join the conversation Follow us Add us as a preferred source on Google Newsletter Subscribe to our newsletter
OpenAI, Google, Microsoft and Anthropic have agreed to support a new way for website owners to stop their content being used to train AI models. The companies are backing a shared opt-out signal in a website’s robots.txt file, which tells AI crawlers whether they're allowed to use its content for training.
Like a modern day Library of Alexandria, the internet is the single largest source of human knowledge that exists. Countless people have spent their lives cataloguing and uploading the entire history of humanity for the betterment of everyone.
AI companies are a tinderbox in this regard. To train and develop better capabilities for their models, AI companies are repeatedly scraping every corner of the internet to create archives of training data. Compounded by changes in search and how people seek information, AI threatens to become the single point of access to information.
Latest Videos FromTechRadarWatch full video here:
But just because you have a website, it doesn’t mean you want it constantly scraped by bots stealing your answers to life’s difficult questions so people can access them elsewhere.
Luckily, you’re not alone. Outrage from users and website hosts across the internet has been fueled into developing methods to keep AI companies from harvesting data from your websites.
Like layering slices of Swiss cheese, these methods work best to keep out scrapers when used in a defense-in-depth manner. But some malicious bots will still find ways to navigate through the holes. Here’s a few ways to keep the honest bots at bay:
1. Robots.txt
This is the crux of OpenAI’s method to honor anti-web scraping markers. The ‘robots.txt’ text file sits in the root directory of your website and provides guidance on how bots should behave. This can be a useful way to provide a digital map to search engine crawlers and can also prevent your site’s servers from becoming overburdened by bot requests.
Get daily insight, inspiration and deals in your inbox
Sign up for breaking news, reviews, opinion, top tech deals, and more.
Contact me with news and offers from other Future brandsReceive email from us on behalf of our trusted partners or sponsors