AI scraping bots are crashing university library websites
Academic libraries say swarms of AI data-scraping bots are overwhelming their sites, locking out students and researchers who compete for access.
What happened: Academic libraries across North America, Europe and Oceania say AI data-scraping bots are slowing or crashing their websites. The University of North Carolina at Chapel Hill reported getting more than 11,000 searches for Finnish music in a single day, from thousands of different internet addresses, and said its catalogue was periodically shutting out students and staff. Librarians describe being bombarded by bots trying to grab as much material as possible, hunting content to train large language models like those behind ChatGPT, Claude and Gemini.
Why it matters: Libraries, museums and archives are easy targets because their material is high-quality, vetted and usually free to access, with no paywall. Experts say ordinary members of the public are now effectively competing with infinite bot traffic just to use a public library site. One researcher calls it an extractive pattern: private companies leaning on public-interest institutions for free data to build commercial products they then sell back to people. Institutions warn the disruption is becoming a disincentive to keep digitizing and sharing collections online.
How it works, plainly: Some bots, like search engines or the Internet Archive, follow robots.txt, a voluntary file telling crawlers what they can and can't take, so they don't overload a site. AI scraping bots often ignore it, sometimes requesting thousands of documents per second in every available format. Because scraping can run from thousands of IP addresses at once, blocking individual addresses no longer works, and defenses like CAPTCHAs or login requirements also end up blocking the legitimate public users libraries are trying to serve.
The rollout: Facing lawsuits and criticism, Anthropic, OpenAI, Google and Microsoft recently said they'd support a single standardized opt-out signal in robots.txt files to block their crawlers. But compliance is voluntary, and it does nothing about third-party data brokers or smaller AI startups running their own scrapers. Groups like Project Gutenberg and GLAM-E are pushing for longer-term fixes, noting that public-domain material could be shared respectfully, with Project Gutenberg already offering a one-click bulk download that bots mostly ignore.