Rho-Perfect: Correlation Ceiling For Subjective Evaluation Datasets

Subjective ratings contain inherent noise that limits the model-human correlation, but this reliability issue is rarely quantified. In this paper, we present $ρ$-Perfect, a practical estimation of the highest achievable correlation of a model on subjectively rated datasets. We define $ρ$-Perfect to be the correlation between a perfect predictor and human ratings, and derive an estimate of the value based on heteroscedastic noise scenarios, a common occurrence in subjectively rated datasets. We show that $ρ$-Perfect squared estimates test-retest correlation and use this to validate the estimate. We demonstrate the use of $ρ$-Perfect on a speech quality dataset and show how the measure can distinguish between model limitations and data quality issues.

💡 Research Summary

The paper addresses a fundamental but often overlooked issue in the evaluation of machine‑learning models that aim to predict subjective human judgments: the inherent noise in human ratings imposes an upper bound on the achievable correlation between model predictions and human scores. To quantify this bound, the authors introduce a new metric called ρ‑Perfect.

Definition and Derivation
A “perfect predictor” is defined as the conditional expectation of the average rating Y given an item X, i.e., Ŷ = E