A Plan For Curating "Obsolete Data or Resources"

Our cultural discourse is increasingly carried in the web. With the initial emergence of the web many years ago, there was a period where conventional mediums (e.g., music, movies, books, scholarly publications) were primary and the web was a supplementary channel. This has now changed, where the web is often the primary channel, and other publishing mechanisms, if present at all, supplement the web. Unfortunately, the technology for publishing information on the web always outstrips our technology for preservation. My concern is less that we will lose data of known importance (e.g., scientific data, census data), but rather that we will lose data that we do not yet know is important. In this paper I review some of the issues and, where appropriate, proposed solutions for increasing the archivability of the web.

💡 Research Summary

The paper “A Plan For Curating ‘Obsolete Data or Resources’” by Michael L. Nelson presents a critical examination of the challenges and future directions for web archiving. It argues that while the web has become the primary channel for our cultural discourse, the technologies for preserving this vast digital record consistently lag behind the rapidly evolving technologies for publishing. The author’s central concern is not the loss of data with recognized, immediate importance (like scientific datasets), but rather the irreversible loss of data whose future significance we cannot yet foresee.

The analysis begins by highlighting a fundamental social and perceptual problem: web archiving remains on the fringes of the broader web community. This is exemplified by a reviewer’s dismissive comment labeling archived content as “obsolete data,” reflecting a widespread lack of compelling use cases that demonstrate immediate value. The author critiques the prevailing “insurance-selling” model of digital preservation, which has failed to generate mainstream enthusiasm.

Technically, the paper traces a core architectural deficiency to the early web’s tight integration with the Unix filesystem. The Unix inode’s limitation—storing last-modified and last-accessed times but not creation time—was inherited by HTTP. This foundational lack of robust temporal semantics means HTTP responses often lack crucial provenance data like when a resource was originally created. The problem is exacerbated by dynamically generated content, where even the “Last-Modified” header becomes meaningless, trapping the web in a “perpetual now.”

The author then critiques the current design paradigm of web archives. Major public archives like the Internet Archive’s Wayback Machine are built as isolated “destinations,” separate from the live web. This requires users to knowingly visit the archive, creating a significant barrier to use. Projects like Memento, which enable temporal browsing integrated with regular browsers, are a step forward but lack a “killer app” to drive widespread adoption. The paper suggests that integrating social or gamified elements—perhaps through watchdog archiving of public figures or collaborative collection-building akin to Pinterest—could energize public participation.

Finally, the paper offers a practical wish list for improving archivability. Key recommendations include: 1) Improving machine-readable time semantics in web services (e.g., ensuring platforms like Twitter expose the correct creation time via HTTP headers, not just in HTML), 2) Developing rich, modern APIs for archives to move beyond error-prone screen scraping and enable developers to build applications, and 3) Moving beyond simple full-text search to provide higher-level services like entity tracking across time, which would better serve scholarly research.

In conclusion, Nelson warns of a Faustian bargain: the web offers unprecedented volume and access to information at the cost of permanence and provenance. Overcoming this requires moving beyond the technical limitations rooted in computing history and re-imagining web archives not as static repositories but as integrated, socially relevant services that provide clear, compelling value to users, scholars, and society at large.