‘The only losers from this will be Cloudflare’s customers. Mainly, publishers’: What businesses can do as Cloudflare readies to categorize AI traffic

Blocking one type of bot traffic could block it all

by · TechRadar

Features By Desire Athow Contributions by Craig Hale Published 27 July 2026

(Image credit: Radware)

Share this article 0 Join the conversation Follow us Add us as a preferred source on Google Newsletter Subscribe to our newsletter

Cloudflare is set to make a major change to huge portions of the internet after standing between websites and automated traffic in a bid to protect publishers and hosters. Soon, the company will divide AI-related traffic into one of three categories – search, agent or training – in a bid to give customers more control over just how much automated traffic they’re prepared to let in.

Beginning September 2026, new Cloudflare domains will block agent and training crawlers by default on pages that display ads, arguing that publishers need human visitors to see those ads to generate a revenue.

But huge challenges stand between now and such a system being rolled out effectively, because many major crawlers like Googlebot are defined as multipurpose and don’t exactly fit into one of those three categories. In the worst case scenario, this could prevent any automated traffic from reaching a site, even if some categories are permitted.

Latest Videos FromTechRadarWatch full video here:

Cloudflare vs. Google: A battle in which publishers could be the actual losers

Worryingly, Googlebot specifically plays a key role in discovering and updating pages in Google Search, and preventing its access could have major implications on content indexing.

Pages that reject Googlebot on the basis that publishers wanted to block at least one of those three categories could gradually lose search visibility, leading to major commercial implications.

Despite the clear disadvantages of blocking automated traffic, many assert that blocking crawlers helps to protect their intellectual property, reduces web hosting infrastructure demands caused by unpredictable traffic spikes, and gives owners more control over how and where their information is used.

Varn Search Marketing Search & Innovation Director Andy Mollison describes the situation as a “high-stakes staring contest” between Cloudflare and Google, but more importantly, one in which publishers are actually the ones who pull the short straw.

Are you a pro? Subscribe to our newsletter

Sign up to the TechRadar Pro newsletter to get all the top news, opinion, features and guidance your business needs to succeed!

Contact me with news and offers from other Future brandsReceive email from us on behalf of our trusted partners or sponsors