Accounting for the Uncertainty in the Evaluation of Percentile Ranks

In a recent paper entitled “Inconsistencies of Recently Proposed Citation Impact Indicators and how to Avoid Them,” Schreiber (2012, at arXiv:1202.3861) proposed (i) a method to assess tied ranks consistently and (ii) fractional attribution to percentile ranks in the case of relatively small samples (e.g., for n < 100). Schreiber’s solution to the problem of how to handle tied ranks is convincing, in my opinion (cf. Pudovkin & Garfield, 2009). The fractional attribution, however, is computationally intensive and cannot be done manually for even moderately large batches of documents. Schreiber attributed scores fractionally to the six percentile rank classes used in the Science and Engineering Indicators of the U.S. National Science Board, and thus missed, in my opinion, the point that fractional attribution at the level of hundred percentiles-or equivalently quantiles as the continuous random variable-is only a linear, and therefore much less complex problem. Given the quantile-values, the non-linear attribution to the six classes or any other evaluation scheme is then a question of aggregation. A new routine based on these principles (including Schreiber’s solution for tied ranks) is made available as software for the assessment of documents retrieved from the Web of Science (at http://www.leydesdorff.net/software/i3).

💡 Research Summary

The paper revisits the two methodological proposals made by Schreiber (2012) for handling citation‑based percentile rankings: (i) a consistent treatment of tied ranks (the “linked‑rank” approach) and (ii) a fractional attribution of scores to the six percentile‑rank classes used in the U.S. National Science Board’s Science and Engineering Indicators. While the linked‑rank solution is sound and aligns with earlier work by Pudovkin and Garfield (2009), the fractional attribution scheme is computationally burdensome, especially when the number of documents exceeds a few dozen. Schreiber’s method requires, for each document, a combinatorial calculation that distributes a document’s contribution across several percentile classes according to the proportion of the class that the document occupies. This quickly becomes infeasible for manual processing and even for many automated pipelines.

The author proposes a fundamentally simpler framework that separates the problem into two linear steps. First, each document is assigned a continuous quantile value q = (r – 0.5)/N, where r is the rank of the document after sorting by citation count and N is the total number of documents. This yields a value in the interval (0, 1) that can be multiplied by 100 to obtain the conventional percentile. Because the calculation is a single arithmetic operation per record, it scales linearly with N and can be performed on datasets of any realistic size in negligible time. Crucially, this step preserves the linked‑rank handling of ties: documents with identical citation counts receive identical q values, guaranteeing consistency.

The second step is an aggregation that maps the continuous quantile to any user‑defined set of percentile classes. For the six classes employed by the Science and Engineering Indicators (0‑1 %, 1‑5 %, 5‑10 %, 10‑25 %, 25‑50 %, 50‑100 %), the mapping is simply a matter of checking which interval the quantile falls into and then applying the class‑specific weight. No fractional splitting of a single document’s score across multiple classes is required; the document contributes its full weight to the single class that contains its quantile. If a more nuanced scheme is desired—e.g., a weighted average across adjacent classes—the same linear quantile can be used as the basis for a straightforward interpolation, but the underlying computation remains linear.

To demonstrate feasibility, the author provides an open‑source Python package (named i3) that implements the entire workflow for records downloaded from the Web of Science. The software automates (1) import of citation data, (2) assignment of linked ranks, (3) calculation of continuous quantiles, (4) user‑defined class boundaries and weight vectors, and (5) generation of summary tables and visualizations. Benchmarks reported in the paper show that a dataset of 10 000 documents is processed in under five seconds on a standard laptop, confirming that the method is suitable for routine institutional assessments, national benchmarking, and policy‑relevant analytics.

By linearising the percentile‑rank problem and decoupling the handling of ties from the aggregation into evaluation classes, the paper eliminates the computational bottleneck that limited Schreiber’s original proposal. The approach retains the methodological rigor of the linked‑rank system while offering a transparent, reproducible, and easily extensible pipeline for citation‑impact evaluation. Consequently, the work makes a significant contribution to the bibliometrics community: it provides a practical solution for dealing with uncertainty in percentile rankings, enhances the scalability of impact assessments, and opens the door for more sophisticated, yet still computationally tractable, evaluation schemes.