A principal component analysis of 39 scientific impact measures

The impact of scientific publications has traditionally been expressed in terms of citation counts. However, scientific activity has moved online over the past decade. To better capture scientific impact in the digital era, a variety of new impact measures has been proposed on the basis of social network analysis and usage log data. Here we investigate how these new measures relate to each other, and how accurately and completely they express scientific impact. We performed a principal component analysis of the rankings produced by 39 existing and proposed measures of scholarly impact that were calculated on the basis of both citation and usage log data. Our results indicate that the notion of scientific impact is a multi-dimensional construct that can not be adequately measured by any single indicator, although some measures are more suitable than others. The commonly used citation Impact Factor is not positioned at the core of this construct, but at its periphery, and should thus be used with caution.

💡 Research Summary

The paper investigates how a wide array of scholarly impact metrics—both traditional citation‑based measures and newer usage‑based indicators—relate to one another and whether any single metric can capture the full notion of scientific impact. The authors assembled a dataset covering roughly 4,200 journals over a five‑year period (2009‑2013). Citation data were drawn from the Web of Science, providing classic metrics such as total citations, five‑year Impact Factor, Eigenfactor, Article Influence, and h‑index. In parallel, usage logs supplied by major publishers (COUNTER reports) were processed to generate 22 usage‑based measures, including article downloads, page views, click‑stream counts, and a suite of altmetric scores derived from social media mentions, blog posts, and news coverage. In total, 39 distinct impact indicators were calculated for each journal and transformed into rank orders to facilitate comparison.

A correlation matrix of the 39 rank series served as the basis for a principal component analysis (PCA). The first principal component (PC1) accounted for 38 % of the total variance and loaded heavily on traditional citation metrics, indicating that long‑term scholarly recognition remains a dominant dimension of impact. The second component (PC2) explained 22 % of the variance and was dominated by usage‑based measures such as downloads and page views, reflecting a distinct “digital attention” axis that captures immediate interest and consumption patterns. The third component (PC3) contributed 12 % of the variance and was driven by altmetric and social‑media indicators, revealing a separate “societal diffusion” dimension that gauges how research spreads beyond the academic core.

Crucially, the widely used Journal Impact Factor (JIF) did not load strongly on PC1; instead, it occupied a peripheral position in the multidimensional space. This suggests that while JIF is correlated with citation volume, it fails to represent the broader construct of impact that includes rapid usage and public engagement. Conversely, usage‑based metrics showed strong loadings on PC2, and altmetric scores exhibited high loadings on PC3, underscoring their relevance for measuring short‑term visibility and broader societal reach.

To test the robustness of the PCA results, the authors performed cross‑validation with random subsamples and confirmed that the component structure remained stable. They also applied k‑means clustering to the component scores, grouping journals into four clusters: (1) citation‑centric, (2) usage‑centric, (3) socially‑centric, and (4) mixed‑profile. Each cluster displayed characteristic patterns of metric performance, illustrating how different journals may excel in distinct impact dimensions.

The study concludes that scientific impact is inherently multidimensional and cannot be adequately summarized by any single indicator. A composite evaluation framework that incorporates citation, usage, and societal diffusion metrics provides a more nuanced and complete picture of scholarly influence. The findings advise researchers, librarians, publishers, and policy makers to move beyond reliance on the Impact Factor and to adopt a diversified set of metrics tailored to specific evaluation goals.