Sampling Issues in Bibliometric Analysis

Bibliometricians face several issues when drawing and analyzing samples of citation records for their research. Drawing samples that are too small may make it difficult or impossible for studies to achieve their goals, while drawing samples that are too large may drain resources that could be better used for other purposes. This paper considers three common situations and offers advice for dealing with each. First, an entire population of records is available for an institution. We argue that, even though all records have been collected, the use of inferential statistics, significance testing, and confidence intervals is both common and desirable. Second, because of limited resources or other factors, a sample of records needs to be drawn. We demonstrate how power analyses can be used to determine in advance how large the sample needs to be to achieve the study’s goals. Third, the sample size may already be determined, either because the data have already been collected or because resources are limited. We show how power analyses can again be used to determine how large effects need to be in order to find effects that are statistically significant. Such information can then help bibliometricians to develop reasonable expectations as to what their analysis can accomplish. While we focus on issues of interest to bibliometricians, our recommendations and procedures can easily be adapted for other fields of study.

💡 Research Summary

The paper addresses a fundamental methodological challenge that bibliometric researchers regularly encounter: determining an appropriate sample size for citation‑record analyses and using that information to guide statistical inference. The authors organize the discussion around three typical scenarios and provide concrete, step‑by‑step recommendations for each.

In the first scenario the researcher has access to the complete population of records for an institution. The authors argue that even when the entire dataset is available, it is still valuable—and often necessary—to treat the data as a sample drawn from a larger, conceptual population. By applying inferential statistics, significance testing, and confidence intervals to randomly selected subsets, researchers can quantify sampling variability, make statements about the precision of estimated indicators (e.g., average citation impact, field‑normalized scores), and provide a statistical basis for policy decisions that compare institutions or track trends over time. The paper stresses that “census” data do not automatically guarantee generalizability; formal inference remains essential for transparent reporting.

The second scenario deals with limited resources that force the researcher to draw a sample. Here the authors advocate a priori power analysis as the cornerstone of sound study design. They walk the reader through the required inputs—anticipated effect size, desired statistical power (commonly 0.80), significance level (α), and the statistical test to be used—and show how to compute the minimum required sample size using standard software such as G*Power, R’s pwr package, or Stata’s power commands. The discussion includes variations for simple random sampling, stratified sampling (e.g., by discipline or publication year), and cluster sampling (e.g., by research group). The authors also address practical complications such as weighting, missing data, and multiple‑comparison adjustments, illustrating how each factor inflates the needed sample size. By performing a power analysis before data collection, researchers can avoid under‑powered studies that would yield a high risk of Type II errors (false negatives) and can allocate resources more efficiently.

The third scenario assumes that the sample size is already fixed—either because data have been collected or because budget constraints are immutable. In this case the paper shows how to reverse the power analysis: given the fixed n, α, and desired power, one can solve for the minimum detectable effect (MDE). This calculation tells the researcher the smallest effect that the study is capable of detecting with statistical significance. If the anticipated effect in the bibliometric context (for example, a 5 % difference in citation impact between two departments) is smaller than the MDE, the authors recommend reconsidering the study design, seeking additional data, or employing more powerful analytical techniques (e.g., hierarchical models, Bayesian methods). They also emphasize reporting the MDE alongside confidence intervals and effect‑size estimates to give readers a realistic sense of the study’s sensitivity.

Throughout the manuscript the authors provide concrete examples drawn from real bibliometric datasets—such as citation counts, co‑authorship networks, and institutional productivity metrics—to demonstrate how the calculations are performed in practice. They include R scripts and screenshots of G*Power output, making the methodology accessible even to researchers with modest statistical training.

Finally, the authors argue that the principles outlined are not limited to bibliometrics. Any field that relies on sampled records—whether in the social sciences, health research, or education—faces the same trade‑off between sample size, statistical power, and resource constraints. By adopting the systematic approach described—using inferential statistics on full populations, conducting a priori power analyses for planned samples, and calculating MDEs for fixed samples—researchers can set realistic expectations, avoid wasted effort, and produce findings that are both statistically robust and practically meaningful. This paper thus serves as a practical guide for improving the methodological rigor of citation‑based studies and for fostering more transparent, evidence‑driven decision‑making across the scholarly ecosystem.