Precision of Treatment Hierarchy: A Metric for Quantifying Certainty in Treatment Hierarchies from Network Meta-Analysis

Network meta-analysis (NMA) is an extension of pairwise meta-analysis which facilitates the estimation of relative effects for multiple competing treatments. A hierarchy of treatments is a useful output of an NMA. Treatment hierarchies are produced using ranking metrics. Common ranking metrics include the Surface Under the Cumulative RAnking curve (SUCRA) and P-scores, which are the frequentist analogue to SUCRAs. Both metrics consider the size and uncertainty of the estimated treatment effects, with larger values indicating a more preferred treatment. Although SUCRAs and P-scores themselves consider uncertainty, treatment hierarchies produced by these ranking metrics are typically reported without a measure of certainty, which might be misleading to practitioners. We propose a new metric, Precision of Treatment Hierarchy (POTH), which quantifies the certainty in producing a treatment hierarchy from SUCRAs or P-scores. The metric connects three statistical quantities: The variance of the SUCRA values, the variance of the mean rank of each treatment, and the average variance of the distribution of individual ranks for each treatment. POTH provides a single, interpretable value which quantifies the degree of certainty in producing a treatment hierarchy. We show how the metric can be adapted to apply to subsets of treatments in a network, for example, to quantify the certainty in the hierarchy of the top three treatments. We calculate POTH for a database of NMAs to investigate its empirical properties, and we demonstrate its use on three published networks.

💡 Research Summary

Network meta‑analysis (NMA) allows simultaneous comparison of many competing interventions, and the resulting treatment hierarchy is usually presented using ranking metrics such as the Surface Under the Cumulative Ranking curve (SUCRA) or its frequentist analogue, the P‑score. While these metrics incorporate the magnitude and uncertainty of effect estimates, they do not provide a single measure of how certain the hierarchy itself is. Consequently, clinicians may over‑interpret a ranking that is actually driven by highly overlapping effect distributions.

The authors introduce a novel hierarchy‑level metric called the Precision of Treatment Hierarchy (POTH). The core idea is to quantify the dispersion of the SUCRA (or P‑score) values across all treatments and to normalise this dispersion by its theoretical maximum, which depends only on the number of treatments n. Let S²(n) denote the variance of the SUCRA values (equivalently, the scaled variance of the expected ranks). In the most uncertain scenario all SUCRA values equal 0.5, giving S²(n)=0; in the most certain scenario the SUCRA values are evenly spaced from 0 to 1, giving the maximal variance S²_max(n) = (n+1)/