Normalization at the field level: fractional counting of citations

Van Raan et al. (2010; arXiv:1003.2113) have proposed a new indicator (MNCS) for field normalization. Since field normalization is also used in the Leiden Rankings of universities, we elaborate our critique of journal normalization in Opthof & Leydesdorff (2010; arXiv:1002.2769) in this rejoinder concerning field normalization. Fractional citation counting thoroughly solves the issue of normalization for differences in citation behavior among fields. This indicator can also be used to obtain a normalized impact factor.

💡 Research Summary

The paper critically examines the field‑level normalization procedures employed by the Centre for Science and Technology Studies (CWTS), particularly the “crown indicators” such as CPP/FCSm and its successor, the Mean Normalized Citation Score (MNCS). The authors argue that these indicators inherit two fundamental problems: (1) the reliance on ISI Subject Categories to delineate scientific fields, and (2) the use of simple averages to normalize citation counts across fields.

First, the authors point out that ISI Subject Categories were designed for information‑retrieval purposes, not for scientometric evaluation. Consequently, the categories suffer from “indexer effects” and lack a solid analytical basis, leading to misclassification of journals and papers. This mis‑alignment can bias field‑level normalization because the same article may be assigned to multiple, overlapping categories, or to a category that does not reflect its true disciplinary context.

Second, the authors contend that normalizing by dividing observed citations by the field’s average citation rate (as done in CPP/FCSm and MNCS) does not adequately control for systematic differences in citation behavior among disciplines. Fields such as mathematics have a low citation density, whereas biomedical sciences exhibit a high density. Using a single world‑average as a benchmark therefore over‑estimates impact in high‑citation fields and under‑estimates it in low‑citation fields.

To address both issues, the paper proposes a fractional counting approach that operates at the level of individual citing articles. Each reference in a citing paper receives a weight equal to 1 divided by the total number of references in that paper. For example, a citation from a mathematics article that lists six references contributes 1/6 of a citation, while a citation from a biomedical article with forty references contributes 1/40. This method directly incorporates the citing author’s citation habits, thereby normalizing for field‑specific citation density without invoking any external classification scheme.

The authors test the new method on a sample of seven researchers drawn from a larger set of 232 scientists evaluated at the Academic Medical Center (AMC) in Amsterdam. Table 1 compares four metrics: (a) the journal‑based CPP/JCSm, (b) the field‑based CPP/FCSm (CWTS 2008), (c) the MNCS, and (d) the newly calculated fractional citation count (Σ c_f). While journal‑based rankings are almost identical (Spearman ρ > 0.99), the field‑based rankings differ substantially. The correlation between the CWTS field indicator and the fractional counting indicator is modest (Spearman ρ ≈ 0.75, not statistically significant), indicating that the traditional field normalization fails to capture the same variance as fractional counting.

Figure 2 presents box‑plots of the fractional citation distributions versus the previously used observed/expected citation ratios. The fractional counts show tighter, more distinct groupings. Post‑hoc statistical tests (Tukey, Bonferroni, and Scheffé) reveal that researchers 1, 2, and 3 form a homogeneous cluster, while researcher 4 is statistically different from them—a distinction that was obscured under the earlier journal‑based normalization.

Beyond descriptive comparisons, the authors emphasize the statistical advantages of fractional counting. Because the method yields a distribution of citation weights for each researcher, standard parametric (t‑tests, Welch’s test) and non‑parametric (Kruskal‑Wallis) procedures can be applied, and confidence intervals or error bars can be plotted. This enables evaluators to assess not only point estimates but also the significance of differences, thereby moving from “raw scores” to “statistically validated performance.”

The paper also argues that fractional counting eliminates the need for a universal “world average” as a reference point. Instead, any reference set—whether the full set of ISI categories, the 60 sub‑fields defined by ECOOM, or a custom collection—can serve as a benchmark. This flexibility is especially valuable in an increasingly interdisciplinary research landscape where field boundaries are porous.

In summary, the authors demonstrate that fractional citation counting provides a conceptually cleaner, statistically robust, and classification‑independent solution to field‑level normalization. It directly adjusts for differences in citation behavior among disciplines, allows rigorous hypothesis testing, and can be extended to normalize journal impact factors by fractionally counting citations to citable items. The proposed approach therefore offers a compelling alternative to the existing CWTS crown indicators and has significant implications for university rankings, funding decisions, and broader science‑policy evaluations.