Revisiting Relative Indicators and Provisional Truths

In the ISSI Newsletter 14(2), Glänzel & Schubert (2018) argue for using "relative indicators"e.g., the Mean Observed Citation Rate relative to the Mean Expected Citation Rate MOCR/MECR (Schubert & Braun, 1986;cf. Vinkler, 1986) -instead of testing citation scores against their expected values using the mean normalized citation score MNCS (Waltman, Van Eck, Van Leeuwen, Visser, & Van Raan, 2011a and b). The authors note our "concern" about using these relative indicators (Opthof & Leydesdorff, 2010;cf. Lundberg, 2007). However, Glänzel & Schubert (2018) state (at p. 47) that they do not wish to "resume the debate but attempt to shed some light on the premises and the context of indicator design in the mirror of the rules of mathematical statistics."

In their discussion of the indicators, Glänzel & Schubert (2018) pay insufficient attention to the differences in terms of the results of a scientometric evaluation. Are the indicators valid and reliable (Lehman et al., 2006)? Our “concern” was never about the relative indicators as mathematical statistics, but about their use in evaluations. From this latter perspective, the division between two averages instead of first normalizing against expected values can be considered as a transgression of the order of mathematical operations by which division precedes addition.

In the case of MOCR/MECR, one first sums in both the numerator and denominator and then divides, as follows:

(1)

In the case of MNCS, one first divides and sums thereafter:

(2)

Eq. 1 has also been called the “Rate of Averages” (RoA) versus the “Average of Rates” (AoR) in the case of Eq. 2 (Gingras & Larivière, 2011).

The “relative indicators” of Eq. 1 were introduced by the Budapest team in the mid-1980s (Schubert & Braun, 1986;Vinkler, 1986). One of these relative indicators-using the field of science as the reference set-has been used increasingly since approximately 1995 as the socalled “crown indicator” (CPP/FCSm)3 by the Leiden unit CWTS (Moed, De Bruin, & Van Leeuwen, 1995). These “relative indicators” are still in use for research evaluations by the ECOOM unit in Louvain headed by Glänzel.

In a vivid debate, Van Raan et al. (2010) first argued that the distinction between RoA and AoR was small and therefore statistically irrelevant. However, both Opthof & Leydesdorff (2010) and Gingras & Larivière (2011) provided examples showing significant differences between the two procedures. Using AoR, one is able to test for the statistical significance of differences in citations among sets of documents. Unlike AoR, RoA comes as a pure number (without error); using this indicator at the time, CWTS and ECOOM invented “rules of thumb” to indicate significance in the deviation from the world standard as 0.5 (Van Raan, 2005) or 0.2 (CWTS, 2008, at p. 7;cf. Schubert & Glänzel, 1983;Glänzel, 1992 and2010). Even if one tries to circumvent the violation of basic mathematical rules by adding brackets to the equations, these conceptual issues remain.

AoR and RoA in the banking world Glänzel & Schubert (2018) refer to a paper published in the arXiv by Matteo Formenti (2014) from the Group Risk Management of the UniCredit Group. In this risk assessment, the author compares default rates of mortgages issued in the years 2008-2011 during the subsequent five years as risks for the bank. The time of default applies to any mortgage that ends before the scheduled date planned by the bank, either because the individual fails to pay or because the mortgage is paid off before the planned date, which also implies less income for a portfolio holder such as a bank.

The problem formulation is different from that of research evaluation using citations:

1. For a bank it does not matter which customers fail to pay the mortgage in the future, as long as the sumtotal of individual positions of customers does not provide a risk for the bank. The sumtotal provides the reference in RoA;

2. Formenti (2014) We do not understand the relation between this example and research evaluations. Are funding agencies distributing money over the scientific community with the aim of avoiding their own bankrupcy?

The new “crown indicator”

In the weeks after voicing our critique (in 2010), the Leiden unit turned up another “crown indicator:” MNCS or the “mean normalized citation score” (Eq. 2; Waltman, van Eck, van Leeuwen, Visser, & van Raan, 2011 a and b). In our response, we expressed our concern about moving too fast-without sufficient debate-to this alternative (Leydesdorff & Opthof, 2011).

Following up on Bornmann & Mutz (2011), we then proposed “to turn the tables one more time” by first specifying criteria for comparing sets of documents in terms of performance indicators independently from specific evaluation contexts and existing infrastructures (Leydesdorff, Bornmann, Mutz, & Opthof, 2011). We formulated these criteria (at pp. 1371f.), as follows:

1. A citation-based indicator must be defined so that the choice of the re

…(Full text truncated)…

This content is AI-processed based on ArXiv data.