Cloudflare Blocks AI Training Bots by Default on New Sites

Cloudflare changed the default behavior for new websites on September 15, 2026. Any domain added to the platform after this date now automatically blocks specific types of AI bots, a shift that impacts how content is accessed by automated systems.

The update distinguishes between different kinds of automated traffic. While standard search indexing remains allowed, bots used for training large language models or acting as real-time agents are blocked by default if the page displays advertisements. This change forces a new standard for how publishers interact with the AI ecosystem.

Automatic blocking targets ad-supported pages

The core of the new policy is a simple trigger: the presence of ads. According to documentation cited by shattered.io, if a page contains advertisements, Cloudflare’s system now classifies the traffic into three buckets. The bucket for search engines stays open, but the buckets for model training and agent activity are closed by default.

This creates a significant trade-off for site owners. While it protects content from being freely consumed by AI models, it also means that any AI assistant relying on these bots will not be able to fetch the page unless the site owner manually changes the settings. The system prioritizes the commercial value of the page over its availability to automated tools.

Existing sites keep their old settings

It is important to note that this change does not retroactively affect every website. Cloudflare specified that the new defaults apply only to domains onboarding after September 15. Websites that were already on the platform retain their previous configurations, meaning no automatic reset occurs for existing users.

This distinction protects sites that have already customized their bot rules. If a publisher had manually allowed or blocked specific crawlers before the update, those settings remain untouched. However, new entrants to the platform start with a stricter posture, effectively changing the baseline for the growing number of sites joining the network.

Monetization shifts the focus from access

Cloudflare pairs these access controls with new monetization features. The company is introducing tools that allow publishers to charge AI companies for crawling their content. This moves the conversation from simple block-or-allow decisions to a billing model.

By combining the default blocks with payment gateways, Cloudflare frames the issue as a commercial one. The implication is that if an AI company wants its bots to access ad-supported content, it must now pay for that access. This creates a new economic layer on top of the technical infrastructure, potentially altering how data is exchanged between publishers and AI developers.