The danger of pseudo science in Informetrics

Two papers have been archived to which this letter is complementary: 1) Opthof and Leydesdorff arxiv:1002.2769 2) Van Raan et al. arxiv:1003.2113 Van Raan at all claims that the order of operations (first dividing then adding) does not apply to citation analysis. In my contribution I discuss a few analogues in Physics and Medicine and argue that in no other field of science where quantities have physical or financial meaning, implying that that numbers have a real unit of measure, it would be allowed to ignore the rule of operations. Hence, the claim of CWTS that the order of operations is not relevant brings studies ignoring this rule as done by CWTS in the category ‘Pseudo Science’.

💡 Research Summary

The paper “The danger of pseudo science in Informetrics” is a methodological critique of the normalization procedures employed by the Centre for Science and Technology Studies (CWTS) in its Leiden rankings of research performance. The author, Jos A.E. Spaan, builds on the earlier work of Opthof and Leydesdorff (2010), who argued that the order of mathematical operations matters when normalizing citation counts by journal citation scores (JCS). Van Raan et al. (2010) responded by claiming that the order of operations is irrelevant for the choice between the two normalization methods. Spaan rejects this claim and demonstrates, through analogies drawn from physics, engineering, and medicine, that ignoring the proper order of operations leads to systematically biased results whenever the quantities involved have physical or financial units.

The central mathematical point is simple: when a ratio is required for each individual observation (e.g., citations per JCS for each paper), the ratio must be computed first for each observation and only then should the average of those ratios be taken. Computing the average numerator and the average denominator separately and then forming the ratio (the “CWTS way”) is mathematically incorrect unless the denominator is constant across observations, which is not the case for journal citation scores. Spaan illustrates this with two concrete analogues.

First, he compares the problem to calculating a health index based on weight (W) and height (L) of a group of people. The correct Quetelet index (or any weight‑to‑height ratio) requires dividing each person’s weight by the square of that person’s height, then averaging the resulting indices. If one instead averages weight and height separately and then divides, the resulting index is about 30 % lower, potentially leading to a false conclusion that the group is under‑nourished and prompting unnecessary dietary interventions.

Second, he uses the calculation of average electrical power consumption across households. Power for each house is the product of voltage and current; the correct average power is obtained by multiplying voltage and current for each house and then averaging the products. Averaging voltages and currents first and then multiplying the two averages yields a value that can be dramatically different from the true average power, violating basic conservation principles.

By analogy, the CWTS method of “average citations divided by average JCS” is shown to underestimate or overestimate the normalized citation impact of individual papers. Using the data presented by Opthof and Leydesdorff, Spaan demonstrates that the CWTS index would be roughly 30 % lower than the Opthof‑Leydesdorff index, which follows the correct per‑paper normalization. This discrepancy is not a trivial statistical fluctuation; it has concrete policy implications because CWTS evaluations are used by institutions such as the Academic Medical Center (AMC) to allocate resources, restructure departments, and even to inform legal actions against perceived unfairness.

Spaan also critiques the reliance on correlation coefficients between the two methods as a defense. A high correlation merely indicates that the two sets of scores move together, not that they are interchangeable or that one is methodologically sound. Scientific methodology demands that the underlying calculation be logically consistent, not just statistically similar.

The paper further argues that the CWTS approach effectively treats scientific quality—a construct without a universally accepted unit of measurement—as if it were a physical quantity subject to the same conservation laws that govern mass, energy, and momentum. By doing so, it blurs the line between rigorous informetric analysis and pseudo‑science. Spaan warns that when managers and policymakers adopt such flawed metrics, they risk making decisions that could jeopardize the careers of excellent researchers and the vitality of interdisciplinary fields.

In conclusion, Spaan calls for a re‑examination of informetric normalization practices. He challenges van Raan and colleagues to provide a genuine scientific example where averaging first and then dividing or multiplying yields a valid result for quantities with physical or financial meaning. Finding none, he asserts that the CWTS method is mathematically unsound and should be abandoned in favor of per‑item normalization. The broader implication is that informetrics must adhere to the same rigorous mathematical standards as any other quantitative science; otherwise, it risks being classified as pseudo‑science with potentially damaging consequences for research evaluation and policy.