Publishers Block Internet Archive Amid AI Concerns, Threatening Web History

Major news publishers have escalated efforts to block the Internet Archive's Wayback Machine, with more than 241 news sites across nine countries restricting access by early 2026[2][8][9]. This widespread blocking action stems from concerns that artificial intelligence (AI) companies are using archived web content for training data[1][2].

This development matters because it creates obstacles for public access to historical web content, jeopardizes journalistic accountability, and hampers the preservation of digital records that researchers and the public rely on[1][2][3][5]. The Internet Archive, which sees itself as a neutral party, warns that these actions undermine its mission to safeguard the internet's history[3][4][12].

What the records show

News organizations, including The New York Times, The Guardian, CNN, and USA Today Co., have implemented measures to prevent the Internet Archive's crawlers from accessing their websites[2][7][8][9][10]. An analysis by AI-detection startup Originality AI identified 23 major news sites blocking ia_archiverbot, the primary crawler for the Wayback Machine[2][9]. Overall, 241 news sites across nine countries disallow at least one of the Internet Archive's four bots, with USA Today Co. accounting for 87% of these blocks[2][9]. The New York Times initiated a “hard block” against archive.org_bot in late 2025, using technical measures that extend beyond standard robots.txt rules[2][6][9][15]. Reddit also announced its own block in August 2025[2]. Publishers fear that AI companies will access their content through the Archive's collections if direct scraping is prevented[15].

The timeline

The trend of publishers blocking the Internet Archive significantly escalated in 2025 and 2026[1]. The New York Times added archive.org_bot to its robots.txt file by the end of 2025[2]. Reddit announced its blocking measure in August 2025[2]. By early 2026, the blocking actions had become widespread, impacting hundreds of news sites[2][9][10].

What changed and for whom

These blocking actions by publishers prevent the Internet Archive from preserving their content, impacting various stakeholders[1][12]. Journalists, researchers, and Wikipedia editors regularly use the Wayback Machine to verify facts, track editorial changes, and access content that might otherwise disappear[2][5][9][11]. For example, USA Today itself published an investigative report made possible by the Archive, even as it blocks the service[11][16]. The removal of content from the historical record means that tools for journalistic accountability are lost, researchers lose potential evidence, and courts lose reference material[12]. This also makes the web more susceptible to historical revisionism[12].

What the subject has said publicly

Mark Graham, director of the Wayback Machine at the Internet Archive, describes his project as “collateral damage caught up in the conflict between AI companies and publishers”[2][3][4][8][9]. He has stated that publishers' concerns about AI scraping via the Archive are “unfounded”[14]. Graham emphasizes that the Archive has implemented its own controls, such as rate limiting and filtering, to prevent large-scale automated extraction by AI companies and other abusive access[3][12][14]. The Internet Archive argues that blocking primarily harms the preservation of the historical record rather than effectively deterring AI scraping, as AI companies could still access archived material through the Archive’s interfaces, which the Archive itself controls[3][12]. Publishers offer differing public statements; a USA Today Co. spokesperson, Lark-Marie Anton, characterized their move as a broader response to “all scraping bots” rather than specifically targeting the Internet Archive[2]. Graham James, a New York Times spokesperson, stated more directly that “The issue is that Times content on the Internet Archive is being used by AI companies in violation of copyright law”[2].

What we could not confirm

The reporting does not quantify how much of the historical record has already been pulled offline due to these blocking actions[2]. The copyright complaints made by publishers against AI scrapers remain unresolved[2]. It is also unconfirmed if an AI company has ever scraped content from the Wayback Machine to train models without permission[15].