Policy documents as sources for measuring societal impact: How often is climate change research mentioned in policy-related documents?

In the current UK Research Excellence Framework (REF) and the Excellence in Research for Australia (ERA) societal impact measurements are inherent parts of the national evaluation systems. In this study, we deal with a relatively new form of societal impact measurements. Recently, Altmetric - a start-up providing publication level metrics - started to make data for publications available which have been mentioned in policy documents. We regard this data source as an interesting possibility to specifically measure the (societal) impact of research. Using a comprehensive dataset with publications on climate change as an example, we study the usefulness of the new data source for impact measurement. Only 1.2% (n=2,341) out of 191,276 publications on climate change in the dataset have at least one policy mention. We further reveal that papers published in Nature and Science as well as from the areas “Earth and related environmental sciences” and “Social and economic geography” are especially relevant in the policy context. Given the low coverage of the climate change literature in policy documents, this study can be only a first attempt to study this new source of altmetric data. Further empirical studies are necessary in upcoming years, because mentions in policy documents are of special interest in the use of altmetric data for measuring target-oriented the broader impact of research.

💡 Research Summary

The paper investigates a novel source of societal impact measurement: mentions of scholarly articles in policy documents, as provided by the altmetrics provider Altmetric. In the context of national research assessment frameworks such as the United Kingdom’s Research Excellence Framework (REF) and Australia’s Excellence in Research for Australia (ERA), which explicitly require evidence of broader impact, the authors explore whether policy‑document citations can serve as a reliable indicator of target‑oriented influence, using climate‑change research as a test case.

A comprehensive dataset of 191,276 climate‑change publications was assembled from major bibliographic databases (Web of Science, Scopus) by applying a broad set of climate‑related keywords. For each record the authors retrieved Altmetric’s “policy mentions” field, which records any instance where the publication is cited in a policy‑relevant source (government reports, international agency documents, white papers, etc.). The matching process identified 2,341 papers that received at least one policy mention, representing only 1.2 % of the entire climate‑change corpus.

Temporal analysis shows a modest upward trend in the proportion of policy‑cited papers after 2015, but the absolute coverage remains low. Disciplinary breakdown reveals that articles classified under “Earth and related environmental sciences” and “Social and economic geography” have the highest policy‑mention rates. This pattern reflects the dual nature of climate‑change challenges: physical‑science research supplies the empirical basis for mitigation and adaptation strategies, while social‑geographic studies provide the contextual understanding required for policy design.

Journal‑level analysis indicates that papers published in the high‑impact multidisciplinary journals Nature and Science are disproportionately represented among the policy‑cited set. The prestige and broad readership of these outlets likely make them preferred sources for policymakers seeking authoritative evidence. In contrast, articles in more specialized or regional journals are seldom referenced in policy documents.

The authors critically assess the limitations of the current policy‑mention dataset. First, Altmetric’s coverage is skewed toward English‑language, internationally visible policy outputs; domestic or sub‑national documents from non‑English speaking countries are under‑represented. Second, the nature of citations in policy texts differs from scholarly referencing: mentions may be narrative, background, or advisory rather than a formal citation, complicating the interpretation of “impact.” Third, the dataset is still in an early stage of development, and its breadth is expected to expand as more policy sources are indexed.

Given these constraints, the study concludes that policy‑document mentions should be treated as a complementary, not standalone, metric for societal impact. Their low prevalence limits the ability to draw robust conclusions about the overall influence of climate‑change research on policy. Nevertheless, the indicator holds promise for capturing the “target‑oriented” dimension of impact that many assessment frameworks seek to evaluate. Future work should aim to broaden the corpus of policy sources, develop methods for contextual analysis of citation intent, and integrate policy mentions with other impact indicators such as patents, industry collaborations, and media attention to construct a more holistic impact assessment model.

In sum, while the current coverage of climate‑change literature in policy documents is modest, the study demonstrates the feasibility of using Altmetric’s policy‑mention data as an additional evidence stream for measuring research influence on policy. Continued data enrichment and methodological refinement will be essential for realizing its full potential in research evaluation and for informing funding agencies and institutions about the tangible societal contributions of scientific work.