How fractional counting affects the Impact Factor: Normalization in terms of differences in citation potentials among fields of science

The ISI-Impact Factors suffer from a number of drawbacks, among them the statistics-why should one use the mean and not the median?-and the incomparability among fields of science because of systematic differences in citation behavior among fields. Can these drawbacks be counteracted by counting citation weights fractionally instead of using whole numbers in the numerators? (i) Fractional citation counts are normalized in terms of the citing sources and thus would take into account differences in citation behavior among fields of science. (ii) Differences in the resulting distributions can be tested statistically for their significance at different levels of aggregation. (iii) Fractional counting can be generalized to any document set including journals or groups of journals, and thus the significance of differences among both small and large sets can be tested. A list of fractionally counted Impact Factors for 2008 is available online at http://www.leydesdorff.net/weighted_if/weighted_if.xls. The in-between group variance among the thirteen fields of science identified in the U.S. Science and Engineering Indicators is not statistically significant after this normalization. Although citation behavior differs largely between disciplines, the reflection of these differences in fractionally counted citation distributions could not be used as a reliable instrument for the classification.

💡 Research Summary

The paper critically examines the conventional ISI Impact Factor (IF) and proposes a fractional counting method to address two major shortcomings: the reliance on the arithmetic mean despite the highly skewed citation distribution, and the lack of comparability across scientific fields due to systematic differences in citation behavior. Traditional IF treats every citation as a whole unit, ignoring the fact that articles in some disciplines (e.g., biomedicine) contain many references and therefore generate a higher “citation potential,” whereas articles in other fields (e.g., mathematics) have far fewer references. Consequently, an IF of 3 in a low‑citation field does not convey the same level of influence as an IF of 3 in a high‑citation field.

The authors’ solution is to weight each citation by the inverse of the citing article’s reference list length. In practice, if a citing paper lists 50 references, each of its citations contributes 1/50 of a citation; if it lists 10 references, each contributes 1/10. This fractional counting normalizes for the citing source’s “citation potential,” thereby adjusting for field‑specific referencing practices. The method can be applied to any document set—individual articles, journals, or groups of journals—allowing statistical testing of differences at various levels of aggregation.

Using the 2008 ISI Web of Science dataset, the authors computed fractionally counted IFs for all indexed journals. They then aggregated journals into the thirteen scientific fields defined by the U.S. Science and Engineering Indicators and performed ANOVA to test whether field means differed significantly after normalization. The results showed that, while conventional IFs exhibited large and statistically significant differences among fields, the fractionally counted IFs did not; the between‑field variance became non‑significant. This indicates that fractional counting effectively removes the bias introduced by divergent citation cultures.

However, the authors caution that the normalized IF should not be taken as a reliable tool for field classification. Even after adjusting for reference length, other disciplinary characteristics—such as research topics, publication frequency, collaborative norms, and the intrinsic quality of cited work—remain unaccounted for. Moreover, fractional counting assumes that all references in a paper are of equal weight, ignoring qualitative differences (e.g., supportive versus critical citations) and the citation impact of the referenced works themselves. The method also inherits the limitations of the underlying citation database, including coverage gaps (especially for non‑English journals and conference proceedings) and potential citation errors.

In conclusion, fractional counting offers a principled way to increase the comparability of IFs across fields by normalizing for the citing source’s reference behavior. It mitigates the inflation of IFs in citation‑rich disciplines and reduces the distortion caused by using a simple arithmetic mean on a skewed distribution. Nonetheless, the approach is not a panacea for research evaluation. Future work should explore hybrid weighting schemes that incorporate the cited article’s own impact (e.g., its own IF or citation count), qualitative citation context, and alternative data sources such as Scopus or Google Scholar. Multi‑dimensional statistical models could further refine field‑level normalization and provide a more nuanced picture of scholarly influence.