AI Crawlers Surge Wikimedia Commons Bandwidth Demand by 50%

The Wikimedia Foundation, the parent body behind Wikipedia and numerous other crowd-sourced knowledge platforms, announced on Wednesday a staggering 50% increase in bandwidth usage for multimedia downloads from Wikimedia Commons since January 2024. This surge, as detailed in a blog post on Tuesday, isn't driven by an uptick in human curiosity, but rather by automated scrapers hungry for data to train AI models.

“Our infrastructure is designed to handle sudden surges in traffic from humans during major events, but the volume of traffic from scraper bots is unmatched and poses increasing risks and costs,” the post explains.

Wikimedia Commons serves as a freely accessible hub for images, videos, and audio files, all available under open licenses or in the public domain.

Delving deeper, Wikimedia revealed that a whopping 65% of the most resource-intensive traffic—measured by the type of content consumed—comes from bots. Yet, these bots account for just 35% of overall pageviews. The discrepancy, according to Wikimedia, stems from how frequently accessed content is cached closer to users, while less popular content, which bots often target, is stored in the more costly "core data center."

“While human readers tend to focus on specific, often similar, topics, crawler bots tend to ‘bulk read’ a larger number of pages and visit less popular ones as well,” Wikimedia noted. “This results in these requests being forwarded to the core datacenter, which significantly increases our resource consumption costs.”

As a result, the Wikimedia Foundation's site reliability team is dedicating substantial time and resources to blocking these crawlers to prevent disruptions for everyday users. This doesn't even touch on the escalating cloud costs the Foundation is contending with.

This scenario is part of a broader trend that's endangering the open internet. Just last month, software engineer and open-source advocate Drew DeVault lamented that AI crawlers are blatantly ignoring “robots.txt” files intended to deter automated traffic. Similarly, Gergely Orosz, known as the "pragmatic engineer," recently voiced his frustration over how AI scrapers from companies like Meta have spiked bandwidth demands for his projects.

While open-source infrastructures are particularly vulnerable, developers are responding with ingenuity and determination. TechCrunch highlighted last week that some tech companies are stepping up. For instance, Cloudflare introduced AI Labyrinth, designed to slow down crawlers with AI-generated content.

Yet, it remains a constant game of cat and mouse, one that might push many publishers to retreat behind logins and paywalls, ultimately harming the open nature of the web we all rely on.