US EditionCloudflare Blocks AI Crawlers by Default, Debuts Pay-Per-Crawl Marketplace
SAN FRANCISCO COUNTY, CALIFORNIA, JUL 1 – Cloudflare now blocks AI crawlers by default and offers a pay-per-crawl option, with over 1 million domains opting in to protect content and generate revenue for creators.
- On Tuesday, Cloudflare launched Pay per Crawl, a private beta marketplace in the US allowing publishers to monetize AI bot access to their websites.
- The launch responds to AI crawlers scraping content without permission, reducing publisher traffic and ad revenue amid fading Google Search visits and rising AI chatbot use.
- Cloudflare’s system lets publishers block AI crawlers by default, set micropayment rates per crawl, and charge different prices for various content types or paths.
- CEO Matthew Prince emphasized their aim to empower creators directly, while a Cloudflare blog highlighted that this approach encourages a transparent marketplace that accurately values original content.
- This model could reshape AI content licensing, but its success depends on AI companies cooperating and on effective crawler detection to avoid disrupting AI training workflows.
93 Articles
93 Articles
How AI can spur a rebirth of media
Reed’s viewThe media business is facing a crisis brought on by artificial intelligence. Since the dawn of the internet, people have had to click links to read articles. Now, they can just ask a chatbot what’s going on in the world.Even if people like Cloudflare co-founder and CEO Matthew Prince have been laudably trying to find a way for publishers to charge the bots, the current business model of internet publishing is under threat.But the adve…
Cloudflare Unveils New Way for Websites to Control and Earn from AI Crawlers
Cloudflare is giving websites a new tool to take back some control as artificial intelligence companies continue to gather info and scrape huge amounts of online content. The new system helps site owners decide if AI bots can access their material, and if so, at what price. For many publishers, this could offer a chance to turn the crawling of their work into something that pays. Websites Are Now Setting the Rules The rise of AI-driven crawlers …
Coverage Details
Bias Distribution
- 48% of the sources lean Left, 48% of the sources are Center
Factuality
To view factuality data please Upgrade to Premium
