Publishing Identifiable Experiment Code And Configuration Is Important, Good and Easy

We argue for the value of publishing the exact code, configuration and data processing scripts used to produce empirical work in robotics. In particular, we recommend publishing a unique identifier for the code package in the paper itself, as a promise to the reader that this is the relavant code. We review some recent discussion of best practice for reproducibility in various professional organisations and journals, and discuss the current reward structure for publishing code in robotics, along with some ideas for improvement.

💡 Research Summary

The paper makes a clear and compelling case that publishing the exact code, configuration files, and data‑processing scripts used in robotics experiments is essential for reproducibility, and that doing so can be both straightforward and beneficial to the research community. It begins by diagnosing the current problem: robotics papers often describe algorithms and results without providing a precise, version‑controlled snapshot of the software stack, hardware parameters, and processing pipelines that generated those results. This omission makes it virtually impossible for other researchers to replicate experiments, especially given the complex interplay of ROS packages, custom drivers, simulation environments, and physical robot specifications.

To address this, the authors propose a concrete workflow centered on a unique identifier (UID) for the code package. The UID can be a DOI assigned by a permanent archive such as Zenodo or Figshare, or an immutable Git commit hash. By embedding this identifier directly in the manuscript (e.g., “Code UID: 10.5281/zenodo.1234567”), authors create an unambiguous link between the paper and the exact software version used. The paper details what should accompany the UID: (1) a complete source tree, (2) explicit dependency specifications (requirements.txt, package.xml, Dockerfile), (3) all configuration files (YAML, launch files, world files), (4) data‑collection and processing scripts, and (5) reproducibility metadata such as random seeds, hardware model numbers, and execution logs. Packaging these elements together—preferably in a version‑controlled repository that is automatically archived with a DOI—ensures that any future reader can retrieve the exact environment needed to rerun the experiments.

The authors then review reproducibility guidelines from major bodies: IEEE Robotics and Automation Society, ACM SIGBOT, and journals such as Nature Methods. While these guidelines uniformly call for open code, data, and permanent identifiers, they fall short on addressing the hardware‑centric nature of robotics. The paper therefore recommends supplementing software releases with (a) simulation equivalents (Gazebo, PyBullet, etc.) that mirror the physical setup, (b) detailed hardware specifications, and (c) virtual machine or container images that encapsulate the entire runtime environment. This “software‑plus‑simulation‑plus‑hardware” package mitigates the black‑box problem and enables both virtual and on‑robot replication.

A substantial portion of the discussion focuses on the incentive structure in academia. Currently, career advancement is measured by citations, conference presentations, and grant dollars, with little direct credit for code sharing. The authors argue that this misalignment discourages researchers from investing the extra effort required for thorough code release. They propose several policy changes: (i) treat code DOIs as citable objects that count toward an author’s citation metrics, (ii) introduce “open‑source contribution points” into grant and promotion evaluations, (iii) require journals to incorporate a code‑review stage where independent reviewers verify that the archived code reproduces the reported results, and (iv) integrate code contributions into ORCID profiles for transparent attribution. By making code a first‑class research output, the community can reward the additional work and encourage a culture of openness.

In conclusion, the paper asserts that publishing an identifiable code package is not a burdensome afterthought but a practical step that dramatically improves reproducibility. When coupled with systematic incentives and standardized documentation of hardware and simulation environments, this practice can raise the overall reliability of robotics research, accelerate scientific progress, and foster greater collaboration across labs. The authors identify future work on developing universal simulation interfaces and automated verification pipelines that could further streamline the reproducibility workflow.