Percentile rank scores are congruous indicators of relative performance, or arent they?

Percentile ranks and the I3 indicator were introduced by Bornmann, Leydesdorff, Mutz and Opthof. These two notions are based on the concept of percentiles (or quantiles) for discrete data. As several definitions for these notions exist we propose one that we think is suitable in this context. Next we show that if the notion of relative congruous indicators is carefully defined then percentile rank scores are congruous indicators of relative performance. The I3 indicator is a strictly congruous indicator of absolute performance.

💡 Research Summary

The paper investigates whether percentile rank scores and the Integrated Impact Indicator (I³) satisfy a property the author calls “congruity,” which is a form of consistency required for performance indicators. Two versions of congruity are defined. For an indicator of average (relative) performance f, two document sets A and B of equal size are considered. If the same document is added to each set, producing A′ and B′, then f is “strictly congruous” if f(A) > f(B) ⇔ f(A′) > f(B′); it is “congruous” if f(A) > f(B) ⇒ f(A′) ≥ f(B′). A similar definition is given for indicators of total (absolute) performance.

The author then formalises percentile rank scores (R) and I³ scores (I₃). For a reference set S, the multiset of citation counts g(S) is partitioned into K disjoint percentile classes. Each class k receives a fixed score x_k. For a set A of N documents, the percentile rank score is

R(A) = (1/N) ∑_{k=1}^{K} x_k · n_A(k)

where n_A(k) is the number of A‑documents in class k. The I³ score drops the division by N:

I₃(A) = ∑_{k=1}^{K} x_k · n_A(k).

The crucial observation is that the class structure (the x_k values) is determined solely by the reference set and does not depend on the evaluated set. Consequently, when the same document belonging to class j is added to both A and B, the term x_j/N (for R) or x_j (for I₃) is added to both numerators, preserving the ordering between the two sets. The paper proves that R is strictly congruous for average performance, while I₃ is strictly congruous for absolute performance, regardless of the sizes of A and B.

The author discusses alternative definitions of percentiles for discrete data. The commonly used Leydesdorff‑Bornmann method defines the percentile of a paper as ⌊100·(#≤ x)/N⌋, which can assign the top paper a score lower than the maximum possible because of the strict “less than” rule. To avoid this, the paper adopts the definition of Beirlant et al. (2005), which uses right‑closed intervals so that the highest citation count always falls into the top percentile and receives the highest score. A detailed numerical example shows how the two approaches differ (e.g., a paper with 18 citations receives a score of 62 under the Beirlant definition but 54 under the Leydesdorff method).

A counterexample is presented to illustrate that percentile rank scores can violate the weaker congruity condition if the reference set changes when a document is added. By adding a zero‑citation paper that also expands the reference set, the percentile class boundaries shift, causing the relative ordering of two groups to reverse. To prevent this, the author refines the definition of congruity (Definition 1a) by requiring that the added document already belong to the intersection of the two reference sets, ensuring that class boundaries remain unchanged.

Finally, the paper notes that the HCP (Highly Cited Publications) indicator is a special case of I₃: if only two classes are defined (highly cited vs. others) and scores of 1 and 0 are assigned, I₃ reduces to a simple count of highly cited papers.

In conclusion, when congruity is carefully defined, percentile rank scores are strictly congruous indicators of relative performance, and the I³ indicator is a strictly congruous indicator of absolute performance. The work clarifies the mathematical foundations of these metrics, highlights the importance of a precise percentile definition, and provides a robust axiomatic basis for their use in research evaluation.