Science-Software Linkage: The Challenges of Traceability between Scientific Knowledge and Software Artifacts

Although computer science papers are often accompanied by software artifacts, connecting research papers to their software artifacts and vice versa is not always trivial. First of all, there is a lack of well-accepted standards for how such links should be provided. Furthermore, the provided links, if any, often become outdated: they are affected by link rot when pre-prints are removed, when repositories are migrated, or when papers and repositories evolve independently. In this paper, we summarize the state of the practice of linking research papers and associated source code, highlighting the recent efforts towards creating and maintaining such links. We also report on the results of several empirical studies focusing on the relationship between scientific papers and associated software artifacts, and we outline challenges related to traceability and opportunities for overcoming these challenges.

💡 Research Summary

The paper “Science‑Software Linkage: The Challenges of Traceability between Scientific Knowledge and Software Artifacts” investigates the fragile and often missing connections between research publications and the software that implements or evaluates the reported ideas. The authors begin by noting that, despite the growing importance of open science, there is no widely accepted standard for embedding persistent links from papers to code and vice‑versa. Consequently, many existing links suffer from “link rot” when pre‑prints disappear, repositories are migrated, or the two artifacts evolve independently.

To quantify the problem, the authors conduct three empirical studies. The first examines GitHub README files that reference academic papers. From roughly 20 000 README files identified by pattern matching, a random sample of 377 was manually inspected; 344 actually cited a paper, and 339 of those citations pointed to openly accessible articles. Domain analysis shows that deep learning, computer vision, and other machine learning topics dominate (about three‑quarters of the sample), while the remaining 20 % span web APIs, biology, chemistry, etc. Only 8 % of the README files showed any change to their citation links over time, and the most common change was a simple replacement of a pre‑print URL with the final publisher URL.

The second study looks at the reverse direction: whether the cited papers contain links back to the corresponding GitHub repositories. Among 136 papers cited in README files, only 62 (≈45 %) provide an “official” repository link, 57 (≈42 %) have no link at all, and a few links lead to dead pages (404). This asymmetry demonstrates that bidirectional traceability is rare, and that researchers often have to search manually to locate the code associated with a paper.

The third study focuses on source‑code comments that reference academic literature. Using a named‑entity recognizer, the authors extracted over 10 000 such comments and manually validated a statistically representative subset of 372 comments, confirming 305 true citations. They classified the knowledge transferred from the paper to the code into eight categories: formulas → source code, pseudocode → source code, textual description → source code, numeric values → hard‑coded constants, scientific findings → hard‑coded rules, documentation only, paper not available, and no transfer. Formulas accounted for 30 % of the transfers, pseudocode 19 %, while 28 % of the comments did not result in any concrete code change. Notably, 13 % of the referenced papers were behind paywalls, preventing any assessment of knowledge transfer.

From these observations the authors derive three major challenges for traceability:

1. Link creation – In traditional software projects, trace links are defined in a traceability plan; in scientific work, links are often added ad‑hoc (e.g., in README or comments). A systematic policy that specifies the purpose, granularity, and format of links is needed.

2. Link identification – Because links appear in heterogeneous locations (README, comments, issue trackers, pull‑request discussions, etc.), automated extraction is difficult without a common metadata schema. Manual effort remains high.

3. Link maintenance – Even when links exist, they are rarely updated when the paper version changes or the repository moves. The authors propose using persistent identifiers that combine DOI (for the paper) and a Git commit hash (for the code) together with automated monitoring tools that alert authors to broken links.

The paper also highlights that most existing links are coarse‑grained, pointing only to a repository’s front page rather than to specific files, functions, datasets, or experimental configurations. This limits reproducibility and hampers fine‑grained knowledge transfer.

In conclusion, the authors argue that robust, bidirectional, and fine‑grained traceability between scientific publications and software artifacts is essential for the open‑science ecosystem. Achieving this requires (a) community‑wide standards for metadata and link formatting, (b) tooling that can automatically discover, validate, and update links across the software development lifecycle, and (c) coordinated policies between publishers, repositories, and research communities to ensure that links remain functional over time. By addressing these challenges, the research community can improve reproducibility, facilitate knowledge reuse, and narrow the “knowledge divide” between practitioners and scholars.