Cloudflare to Block AI Crawlers From ad-Supported Webpages by Default
Cloudflare will default new sites to block mixed-use crawlers from ad-supported pages and will pay publishers through new partnerships and a dashboard.
- On Wednesday, Cloudflare announced that starting September 15, 2026, it will default to blocking mixed-use AI crawlers from accessing ad-supported customer websites, including those on its free tier.
- Many publishers feel uncompensated for content used to train AI models, as mixed-use crawlers often combine search indexing with data harvesting; Cloudflare stated the change encourages crawlers to separate search traffic from agent use.
- The company is also evolving its "Pay Per Crawl" marketplace into "Pay Per Use," allowing publishers to charge AI companies when their content creates value, testing this model with Ceramic.ai and You.com.
- Cloudflare is launching a Business Insights Dashboard and Answer Engine Optimization tools to provide visibility into bot traffic, letting publishers track how their content appears in AI-generated responses.
- CEO Matthew Prince stated that with non-human traffic now the majority online, the company must act faster to build a sustainable ecosystem, positioning Cloudflare as the infrastructure layer managing access, attribution, and payments between publishers and AI.
19 Articles
19 Articles
Cloudflare wants to build the economic layer of the AI web
AI has changed the web right before our eyes. With Google’s AI Overviews doing the heavy lifting, publications that once owned the first page of search results are being replaced by summaries. Readers get their answer without ever clicking through. Much of the traffic has simply stopped. Cloudflare on Wednesday announced a slew of updates for publishers who are facing this new reality. From new crawler classifications and analytics dashboards to…
Cloudflare's new policy pushes AI companies to pay for publishers' content
Cloudflare is giving AI companies until September 15 to separate web crawlers used for search from those used for AI training and agents, or risk being blocked by default on many publisher sites.
Cloudflare to block cynical search-and-scrape bots from ad-supported web pages
Cloudflare on Wednesday said it will soon prevent mixed-use crawlers from accessing ad-supported customer websites by default, part of its ongoing efforts to give site publishers more control over how they engage with AI services. Apple, Google, and Microsoft's Bing operate crawlers that could fall afoul of Cloudflare's decision, although each of the tech giants offers an AI opt-out that may allow them to escape sanctions. Web crawlers make auto…
Coverage Details
Bias Distribution
- 57% of the sources are Center
Factuality
To view factuality data please Upgrade to Premium
