The Impact of LLMs on Online News Consumption and Production

Large language models (LLMs) change how consumers acquire information online; their bots also crawl news publishers’ websites for training data and to answer consumer queries; and they provide tools that can lower the cost of content creation. These changes lead to predictions of adverse impact on news publishers in the form of lowered consumer demand, reduced demand for newsroom employees, and an increase in news “slop.” Consequently, some publishers strategically responded by blocking LLM access to their websites using the robots.txt file standard. Using high-frequency granular data, we document four effects related to the predicted shifts in news publishing following the introduction of generative AI (GenAI). First, we find a moderate decline in traffic to news publishers occurring after August 2024. Second, using a difference-in-differences approach, we find that blocking GenAI bots can be associated with a reduction of total website traffic to large publishers compared to not blocking. Third, on the hiring side, we do not find evidence that LLMs are replacing editorial or content-production jobs yet. The share of new editorial and content-production job listings increases over time. Fourth, regarding content production, we find no evidence that large publishers increased text volume; instead, they significantly increased rich content and use more advertising and targeting technologies. Together, these findings provide early evidence of some unforeseen impacts of the introduction of LLMs on news production and consumption.

💡 Research Summary

The paper investigates the early impact of generative AI (GenAI) and large language models (LLMs) on the online news ecosystem by combining high‑frequency data on website traffic, page structure, and labor market activity. Using daily domain‑level visit estimates from SimilarWeb (Oct 2022–Jul 2025) and household‑level browsing data from the Comscore Web‑Behavior Panel, the authors first identify structural breaks in traffic patterns with a change‑point detection algorithm (Killick et al., 2012). Two prominent breaks emerge: November 2023 and August 2024, with the latter associated with a sustained 13.2 % decline in visits to news sites relative to a control group of the top 100 retail sites, as estimated by a synthetic difference‑in‑differences (SDID) model (Arkhangelsky et al., 2021).

The second contribution evaluates the effect of blocking LLM crawlers via robots.txt. By tracking when each publisher added a “Disallow” rule for GPT‑related bots (using HTTP Archive snapshots) and exploiting the staggered adoption across publishers, a staggered difference‑in‑differences design (Callaway & Sant’Anna, 2021) compares “blocking” publishers to those not yet blocking and to never‑blocking outlets. The analysis finds a roughly 10 % reduction in log weekly visits (SimilarWeb) during the 12 weeks after a block is implemented. Replication with Comscore’s human‑only panel shows a noisier but directionally consistent decline, suggesting that the traffic loss is not limited to automated bot visits but may also affect genuine audience demand.

Third, the paper examines labor market responses using job postings from Revelio Labs. Contrary to the hypothesis that LLMs would substitute editorial and content‑production labor, the share of new postings for editorial and content‑production occupations actually rises over the sample period, indicating no short‑run displacement and possibly a reallocation toward new skill sets required for richer multimedia content.

Finally, the authors analyze changes in content format and page composition. Leveraging HTML metadata from the HTTP Archive and URL counts from the Internet Archive’s Wayback Machine, they find no significant increase in the number of text‑heavy sections or article URLs. Instead, there is a substantial rise in interactive elements (+68.1 % relative to retail sites) and advertising/targeting components (+50.1 %). Growth is concentrated in image‑related URLs, reflecting a shift toward “rich” content rather than sheer text volume.

Overall, the study documents four empirical facts: (1) a moderate, delayed decline in news‑site traffic after August 2024; (2) blocking LLM crawlers appears to exacerbate traffic loss rather than merely remove bot traffic; (3) no evidence of short‑run editorial job cuts; and (4) a clear pivot toward richer, multimedia‑heavy pages. These findings suggest that while LLMs are not yet a direct substitute for traditional news production, they are prompting publishers to adjust access controls, hiring composition, and content formats—adjustments that can have unforeseen negative consequences for audience reach. The paper calls for further research on long‑term revenue effects, user experience implications, and the evolving legal‑regulatory landscape surrounding AI‑driven content reuse.