Alternative metrics in scientometrics: A meta-analysis of research into three altmetrics

Alternative metrics are currently one of the most popular research topics in scientometric research. This paper provides an overview of research into three of the most important altmetrics: microblogging (Twitter), online reference managers (Mendeley and CiteULike) and blogging. The literature is discussed in relation to the possible use of altmetrics in research evaluation. Since the research was particularly interested in the correlation between altmetrics counts and citation counts, this overview focuses particularly on this correlation. For each altmetric, a meta-analysis is calculated for its correlation with traditional citation counts. As the results of the meta-analyses show, the correlation with traditional citations for micro-blogging counts is negligible (pooled r=0.003), for blog counts it is small (pooled r=0.12) and for bookmark counts from online reference managers, medium to large (CiteULike pooled r=0.23; Mendeley pooled r=0.51).

💡 Research Summary

The paper conducts a systematic meta‑analysis of the relationship between three widely discussed alternative metrics—Twitter mentions, blog citations, and bookmarks from online reference managers (Mendeley and CiteULike)—and traditional citation counts. The authors begin by framing altmetrics as a response to the well‑known limitations of citation‑based evaluation, emphasizing that social media platforms capture rapid, societal attention while reference‑manager data may reflect scholarly reading and intent to cite. A comprehensive literature search covering publications from 2010 to 2023 identified 44 empirical studies that reported Pearson correlation coefficients (r) between each altmetric and citation counts for the same set of articles, together with sample sizes. Inclusion criteria required that both altmetric and citation data be derived from the same article cohort and that the original studies provide sufficient statistical detail.

For each altmetric, the authors extracted the reported r values and applied standard meta‑analytic techniques. Heterogeneity was assessed using Cochran’s Q and the I² statistic; random‑effects models were employed when I² exceeded 75 %, otherwise fixed‑effects models were used. Meta‑regression examined potential moderators such as publication year, disciplinary field (natural vs. social sciences), and data source, but none reached statistical significance.

The pooled results reveal stark differences among the metrics. Twitter shows virtually no relationship with citations (pooled r = 0.003, 95 % CI −0.02 to 0.03, I² = 82 %). This supports the view that Twitter activity primarily reflects public or media interest rather than scholarly impact. Blog mentions exhibit a modest positive correlation (pooled r = 0.12, 95 % CI 0.07 to 0.17, I² = 68 %), suggesting that blog posts—often authored by experts or science communicators—may occasionally highlight work that later receives citations. The strongest association is found for Mendeley bookmarks (pooled r = 0.51, 95 % CI 0.44 to 0.58, I² = 55 %). This medium‑to‑large effect indicates that saving an article in a personal reference manager is a good proxy for scholarly interest and a predictor of future citation. CiteULike, by contrast, yields a smaller but still positive correlation (pooled r = 0.23, 95 % CI 0.15 to 0.31, I² = 71 %), likely reflecting its more limited user base and differing data‑capture practices.

The discussion interprets these findings in the context of research evaluation. The authors argue that while Twitter and, to a lesser extent, blogs can enrich understanding of societal impact, they should not be used as sole indicators of scholarly quality. Mendeley’s robust correlation makes it a promising complementary metric for early‑stage impact assessment, especially when citation windows are short. CiteULike’s weaker performance underscores the importance of coverage and user demographics in altmetric reliability.

Limitations are acknowledged: potential publication bias, reliance on bivariate correlations (which cannot establish causality), variability in data collection timestamps, and insufficient granularity to capture field‑specific dynamics. The paper calls for future work employing multivariate models, longitudinal designs, and the inclusion of emerging altmetrics such as policy document citations or dataset downloads. Ultimately, the authors conclude that altmetrics are heterogeneous tools; their utility depends on the specific metric, disciplinary context, and evaluation purpose. A balanced, multi‑metric framework that integrates both traditional citations and carefully selected altmetrics is recommended for a more nuanced assessment of research influence.