Human Perception of Performance

Humans are routinely asked to evaluate the performance of other individuals, separating success from failure and affecting outcomes from science to education and sports. Yet, in many contexts, the metrics driving the human evaluation process remain unclear. Here we analyse a massive dataset capturing players’ evaluations by human judges to explore human perception of performance in soccer, the world’s most popular sport. We use machine learning to design an artificial judge which accurately reproduces human evaluation, allowing us to demonstrate how human observers are biased towards diverse contextual features. By investigating the structure of the artificial judge, we uncover the aspects of the players’ behavior which attract the attention of human judges, demonstrating that human evaluation is based on a noticeability heuristic where only feature values far from the norm are considered to rate an individual’s performance.

💡 Research Summary

The paper investigates how human judges evaluate soccer player performance by analyzing a massive dataset of 760 Italian Serie A matches from the 2015/16 and 2016/17 seasons. For each player‑game pair the authors extracted 150 technical features (passes, shots, tackles, etc.) from event logs, standardized them, and paired them with the numeric ratings (0–10 in 0.5‑point steps) assigned by three leading Italian sports newspapers (Gazzetta dello Sport, Corriere dello Sport, and Tuttosport). The rating distributions are highly similar across newspapers, centered around 6, and show a strong Pearson correlation (r ≈ 0.76) with a typical inter‑judge RMSE of 0.5, indicating overall agreement but also occasional large discrepancies.

First, the authors quantify the relationship between technical similarity (Minkowski distance between feature vectors) and rating difference, confirming that more similar performances tend to receive closer ratings. To probe the underlying decision process, they train a machine‑learning “artificial judge” using only technical features (model M_P). This model achieves a Pearson correlation of 0.55 and RMSE of 0.60 when predicting human ratings—substantially lower than the human‑human agreement, suggesting that technical data alone cannot fully explain the judges’ decisions.

Next, they augment the feature set with contextual information: player age, nationality, club, expected match outcome from bookmakers, actual result, and home/away status. The enriched model (M_{P+C}) markedly improves performance (r ≈ 0.68, RMSE ≈ 0.54) and reduces the Kolmogorov‑Smirnov distance between predicted and real rating distributions. Error analysis reveals that the remaining mismatches are mainly extreme outliers (ratings > 7 or < 5) that involve rare events (e.g., a player becoming the all‑time top scorer) not captured by the available data.

Feature‑importance analysis, performed separately for the four player roles (goalkeeper, defender, midfielder, forward), shows a striking role‑dependent pattern. Goalkeepers and forwards are judged primarily on direct technical metrics (saves, goals), whereas defenders and midfielders receive higher weight from collective contextual variables such as team goal difference and match outcome. Moreover, the predictive power of the model saturates after about 20 of the most important features, indicating that human judges rely on a small subset of salient cues rather than the full high‑dimensional feature space.

The authors formalize this observation as a “noticeability heuristic”: judges first attend to a limited set of features that stand out—values far from the norm—and base their rating on these conspicuous deviations. When average ratings (5.5–6.5) are examined, most feature values cluster around the mean; only for high (> 7) or low (< 5) ratings do specific features deviate markedly, driving the judgment. This heuristic explains why human evaluation appears simple and robust despite the underlying complexity of the sport.

In summary, the study demonstrates that human performance evaluation in soccer is not a comprehensive statistical assessment but a cognitively economical process that emphasizes a few noticeable, often context‑driven cues. The findings have broader implications for any domain where expert judgment is used (education, science, arts), highlighting potential sources of bias and suggesting that machine‑learning proxies can both model and illuminate human decision‑making. Future work could incorporate richer psychological and media‑related variables and explore real‑time decision‑support tools based on the artificial judge framework.