Stochastically Dominant Peer Prediction

Eliciting reliable human feedback is essential for many machine learning tasks, such as learning from noisy labels and aligning AI systems with human preferences. Peer prediction mechanisms incentivize truthful reporting without ground truth verification by scoring agents based on correlations with peers. Traditional mechanisms, which ensure that truth-telling maximizes the expected scores in equilibrium, can elicit honest information while assuming agents’ utilities are linear functions of their scores. However, in practice, non-linear payment rules are usually preferred, or agents’ utilities are inherently non-linear. We propose stochastically dominant truthfulness (SD-truthfulness) as a stronger guarantee: the score distribution of truth-telling stochastically dominates all other strategies, incentivizing truthful reporting for a wide range of monotone utility functions. Our first observation is that no existing peer prediction mechanism naturally satisfies this criterion without strong assumptions. A simple solution – rounding scores into binary lotteries – can enforce SD-truthfulness, but often degrades sensitivity, a key property related to fairness and statistical efficiency. We demonstrate how a more careful application of rounding can better preserve sensitivity. Furthermore, we introduce a new enforced agreement (EA) mechanism that is theoretically guaranteed to be SD-truthful in binary-signal settings under mild assumptions, and empirically achieves the highest sensitivity among all known SD-truthful mechanisms.

💡 Research Summary

The paper tackles a fundamental limitation of existing peer‑prediction mechanisms: they guarantee truthfulness only when agents’ utilities are linear in the scores they receive. In many real‑world applications—such as threshold bonuses, grade‑based rewards, or intrinsic preferences—the utility functions are monotone but non‑linear, so the standard equilibrium notion (maximizing expected score) no longer ensures honest reporting.

To address this, the authors introduce stochastic‑dominance truthfulness (SD‑truthfulness). A mechanism is SD‑truthful if, when all other agents report truthfully, the distribution of an agent’s score under the truthful strategy first‑order stochastically dominates the distribution under any unilateral deviation. By a classic result in microeconomics, this dominance is equivalent to guaranteeing higher expected utility for every increasing utility function, thus covering all monotone non‑linear preferences.

The paper first shows that none of the well‑known peer‑prediction mechanisms—Output Agreement (OA), Peer Truth Serum (PTS), Correlated Agreement (CA), and Matching Agreement (MA)—are naturally SD‑truthful under arbitrary information structures. OA is SD‑truthful only under a strong self‑dominating signal assumption; PTS and CA require self‑predicting signals but still fail because their individual scores take multiple values, breaking the stochastic‑dominance condition.

The authors then propose two generic “rounding” transformations that can turn any bounded‑score mechanism into an SD‑truthful one.

1. Direct rounding reduction normalizes the original score to a probability λ∈