US EditionAI Site Perplexity Uses “Stealth Tactics” to Flout No-Crawl Edicts, Cloudflare Says
INTERNET-WIDE (TENS OF THOUSANDS OF DOMAINS), AUG 4 – Cloudflare's report details how Perplexity used stealth bots to evade blocks on over 10,000 domains, raising concerns about AI data use and web content protection.
- On August 4, 2025, Cloudflare published research stating it observed Perplexity’s use of undeclared crawlers to scrape sites that blocked its bots via robots.txt files.
- After websites blocked Perplexity via robots.txt files and Web application firewalls, Cloudflare researchers say the stealth bots used rotated IPs and altered user agents to evade detection.
- Monitoring logs revealed over 10,000 domains and millions of requests per day by Perplexity's stealth bots, Cloudflare said.
- In response, Cloudflare de-listed Perplexity's bots from its verified list and added heuristics to block stealth crawling.
- Looking ahead, stakeholders anticipate more publishers will adopt AI crawler blocking defenses, potentially accelerating calls for legal frameworks for web scraping, as Cloudflare's default blocking gains traction.
Insights by Ground AI
Does this summary seem wrong?
15 Articles
15 Articles
Cloudflare Says Perplexity's AI Bots Ignore No-Crawl Directives
Global cloud platform company Cloudflare has accused Perplexity AI of using “stealth, undeclared crawlers” to intentionally ignore websites’ no-crawl directives. In other words, Perplexity is getting around regulations aimed at stopping its AI web crawlers from receiving access to certain websites, according to this new report. For AI critics, this news might serve as another indicator that AI tools are built off the backs of unauthorized data. …
Coverage Details
Total News Sources15
Leaning Left1Leaning Right0Center4Last UpdatedBias Distribution80% Center
Bias Distribution
- 80% of the sources are Center
80% Center
L 20%
C 80%
Factuality
To view factuality data please Upgrade to Premium
