Devs say AI crawlers dominate traffic, forcing blocks on entire countries

Devs say AI crawlers dominate traffic, forcing blocks on entire countries

Software developer Xe Iaso reached a breaking point earlier this year when aggressive AI crawler traffic from Amazon overwhelmed their Git repository service, repeatedly causing instability and downtime. Despite configuring standard defensive measures—adjusting robots.txt, blocking known crawler user-agents, and filtering suspicious traffic—Iaso found that AI crawlers continued evading all attempts to stop them, spoofing user-agents and cycling …

AI-crawlers dominate traffic and react to blockages with skilled user agents. The open source scene resists it. (Open Source, AI)

AI Data Poisoning - Schneier on Security

Cloudflare has a new feature—available to free users as well—that uses AI to generate random pages to feed to AI web crawlers: Instead of simply blocking bots, Cloudflare’s new system lures them into a “maze” of realistic-looking but irrelevant pages, wasting the crawler’s computing resources. The approach is a notable shift from the standard block-and-defend strategy used by most website protection services. Cloudflare says blocking bots someti…

Developers say aggressive AI crawlers are overwhelming open-source infrastructure and disproportionately burdening open-source projects

Benj Edwards / Ars Technica: Developers say aggressive AI crawlers are overwhelming open-source infrastructure and disproportionately burdening open-source projects  —  Software developer Xe Iaso reached a breaking point earlier this year when aggressive AI crawler traffic from Amazon overwhelmed their Git repository service …

Exploration robots powered by artificial intelligence are massively invading the web, now accounting for 28% of the traffic traditionally dominated by Googlebot. This new generation of crawlers, more greedy and less respectful of server resources, is testing online infrastructures and is pushing many publishers to take radical measures of protection.

Open Source Devs Say AI Crawlers Dominate Traffic, Forcing Blocks On Entire Countries

An anonymous reader quotes a report from Ars Technica: Software developer Xe Iaso reached a breaking point earlier this year when aggressive AI crawler traffic from Amazon overwhelmed their Git repository service, repeatedly causing instability and downtime. Despite configuring standard defensive measures -- adjusting robots.txt, blocking known crawler user-agents, and filtering suspicious traffic -- Iaso found that AI crawlers continued evading…