Challenges in Selecting Software to be Reused

This is a position paper for Sharing, Re-Use and Circulation of Resources in Cooperative Scientific Work, a CSCW'14 workshop. It discusses the role of software in NSF’s CIF21 vision and the SI2 program, which is intended to support that goal. SI2 primarily supports software projects that are proposed in response to solicitations, and some of the criteria used by the peer-reviewers and by NSF in evaluating these projects depend on predicting scientific impact. This paper discusses some ideas on how the prediction of scientific impact can be improved.

💡 Research Summary

The paper, presented at the CSCW’14 workshop on Sharing, Re‑Use and Circulation of Resources in Cooperative Scientific Work, is a position statement that examines the difficulties inherent in selecting software for reuse within the framework of the National Science Foundation’s (NSF) CIF21 vision and its Software Infrastructure for Sustained Innovation (SI2) program. CIF21 envisions a future scientific ecosystem in which high‑performance computing, data‑intensive workflows, and collaborative tools are seamlessly integrated, and it explicitly identifies reusable software as a cornerstone of that ecosystem. The SI2 program operationalizes this vision by providing sustained funding to software projects that respond to specific solicitations. However, the success of SI2 projects hinges on the ability of reviewers and NSF program officers to predict the scientific impact that a proposed software system will generate once it is released to the community.

The authors argue that current impact‑prediction practices are fundamentally limited. Reviewers rely on qualitative judgments about the novelty of the proposed solution, the size of the anticipated user base, and the alignment with NSF’s strategic priorities. These judgments are prone to bias, suffer from incomplete information, and often fail to capture the complex, long‑term dynamics that determine whether a piece of software will be widely adopted, cited, or integrated into other research pipelines. Empirical evidence from past SI2 awards shows a mismatch between projected and realized impact: many projects that received high scores in the proposal stage later struggled with low download counts, minimal community engagement, and insufficient maintenance resources.

To address these shortcomings, the paper proposes three interlocking strategies. First, it calls for the creation of a quantitative impact‑prediction model built on historical data from previously funded software. Metrics such as download statistics, GitHub activity (stars, forks, issue resolution time), citation counts of associated publications, and user‑survey results would be aggregated into a structured database. Machine‑learning techniques—regression, random forests, or gradient‑boosted trees—could then be trained to forecast future impact based on a set of observable features extracted from new proposals. Second, the authors recommend that proposals include a detailed “reuse plan” that specifies modular architecture, standardized APIs, comprehensive documentation, licensing choices, and concrete outreach activities (tutorials, workshops, community forums). By making the reuse strategy explicit, reviewers can assess the feasibility of widespread adoption more objectively. Third, the paper suggests the adoption of a standardized impact‑assessment checklist for reviewers and NSF staff. The checklist would enumerate criteria such as code quality (test coverage, continuous‑integration practices), community health indicators (number of active contributors, response time to issues), sustainability provisions (funding model beyond the award period), and alignment with broader scientific workflows. This would reduce inter‑reviewer variability and increase the transparency of the decision‑making process.

Beyond technical considerations, the authors emphasize the importance of social and organizational factors in software reuse. Successful reuse often depends on the existence of an active user community, training resources, and responsive support structures. Consequently, the predictive model should incorporate non‑technical variables—team size and experience, prior community‑building activities, frequency of workshops, and the presence of dedicated support staff. By integrating these dimensions, the model would better capture the “human” side of software sustainability, which is frequently overlooked in purely technical assessments.

In conclusion, the paper diagnoses a critical gap in NSF’s current approach to funding reusable scientific software: the reliance on subjective, low‑resolution impact forecasts. It offers a roadmap that blends data‑driven prediction, explicit reuse planning, and standardized evaluation tools to improve the reliability of impact assessments. Implementing these recommendations would not only increase the likelihood that SI2‑funded projects achieve their intended scientific reach but also advance the broader CIF21 goal of building a resilient, collaborative computational infrastructure for future research.