Scorio.jl: A Julia package for ranking stochastic responses

Scorio.jl is a Julia package for evaluating and ranking systems from repeated responses to shared tasks. It provides a common tensor-based interface for direct score-based, pairwise, psychometric, voting, graph, and listwise methods, so the same benchmark can be analyzed under multiple ranking assumptions. We describe the package design, position it relative to existing Julia tools, and report pilot experiments on synthetic rank recovery, stability under limited trials, and runtime scaling.

💡 Research Summary

Scorio.jl is a Julia library designed to simplify the ranking of systems when the evaluation data consist of repeated responses to a shared set of tasks. The core data structure is a three‑dimensional integer tensor R∈{0,…,C}^{L×M×N}, where L denotes the number of systems, M the number of tasks (or questions), and N the number of trials (or samples) per task. By treating this tensor as the canonical input, Scorio.jl can internally generate the various “views” required by a wide range of ranking families—pointwise averages, pairwise win‑count matrices, per‑task orderings, latent‑trait estimates, voting tallies, and weighted comparison graphs—without the user having to reshape or reformat the data.

The package provides a unified API: a single function call such as r = bayes(R; return_scores=true) returns both a score vector and a rank vector, and the same tensor can be passed to other methods (bradley_terry, rasch_mml, pagerank, etc.) with identical syntax. Internally, Scorio.jl converts the tensor into the appropriate representation for each method: means for score‑based methods, win/tie counts for paired‑comparison models, sorted per‑task lists for voting and listwise models, and a weighted adjacency matrix for graph‑based algorithms. A dedicated tie‑aware ranking layer then maps continuous scores to several ranking conventions (competition, dense, ordinal, fractional), making the handling of ties explicit and interchangeable.

Scorio.jl also supports explicit priors for MAP‑style estimators. Users can choose from Gaussian, Laplace, Cauchy, uniform, custom, or empirical priors, enabling the incorporation of knowledge from previous benchmark runs or domain expertise directly into the ranking process.

The library aggregates a broad set of ranking families:

• Evaluation‑based: simple averages, Bayesian score aggregation, various Pass@k variants.
• Sequential rating: Elo, Glicko, TrueSkill, which treat each trial as a match.
• Paired‑comparison: Bradley‑Terry, Davidson, Rao‑Kupper, Thompson sampling, Bayesian MCMC.
• Item‑Response Theory (IRT): Rasch, 2‑PL, 3‑PL, maximum‑likelihood (MML) and dynamic IRT.
• Voting rules: Borda, Copeland, Schulze, Ranked Pairs, Kemeny‑Young, Nanson, Baldwin, Majority Judgment.
• Graph and spectral methods: PageRank, Rank Centrality, AlphaRank, Nash meta‑game solver, SerialRank, HodgeRank.
• Listwise choice models: Plackett‑Luce, Davidson‑Luce, Bradley‑Terry‑Luce.

The authors evaluate Scorio.jl on three fronts. First, they test synthetic rank recovery using a Rasch‑style latent‑ability model with L = 11 systems, M = 500 tasks, and varying numbers of trials N ∈ {1,2,4,8,16,32}. Most methods achieve high Kendall τ_b (0.96–0.98) even with a single trial and improve to τ_b ≈ 0.991 with 32 trials, while Elo lags behind (τ_b ≈ 0.62) due to its sequential update nature. No method perfectly recovers an intentionally inserted tie, highlighting the difficulty of exact tie detection.

Second, they assess stability under limited trials by fixing a “reference” ranking obtained with N_max = 64 trials (using the Bayesian method) and recomputing rankings with only the first n ∈ {1,2,4,8,16,32} trials. Score‑based methods (average, Bayesian, g‑Pass@k), PageRank, and BT‑Davidson already achieve τ_b ≈ 0.97 and 100 % top‑1 agreement after a single trial; Rasch starts lower (τ_b ≈ 0.956) but catches up by n = 8. Elo remains unstable (τ_b ≈ 0.58, top‑1 agreement ≈ 40 %). This demonstrates that method choice matters most when data are scarce.

Third, they benchmark runtime scaling across system counts L ∈ {4,8,16}, task counts M ∈ {100,500,1000}, and trial counts N ∈ {1,4}. Simple score‑based and graph methods (average, Bayesian, Borda, PageRank, AlphaRank) consistently run under 2 ms per call. Rasch (MML implementation) is the most expensive, taking up to ~9 s and 12.8 GB of memory for (L, M, N) = (16, 1000, 1). Kemeny‑Young, due to its NP‑hard combinatorial nature, is only feasible for L ≤ 8 and costs 0.34–0.42 s. These results guide practitioners toward fast baseline methods for routine benchmarking while reserving more costly psychometric models for deeper analysis.

From a software‑engineering perspective, Scorio.jl leverages Julia’s multiple dispatch and high‑performance dense arrays to present a single, type‑stable interface for a heterogeneous set of algorithms. Two‑dimensional inputs are automatically promoted to three‑dimensional tensors, allowing single‑trial benchmarks to follow the same code path as multi‑trial experiments. The package also provides utilities for converting score vectors into multiple tie‑aware rank views, facilitating sensitivity analyses of tie‑handling policies.

The authors argue that many research domains—large language model evaluation (e.g., Pass@k), stochastic software benchmarking, educational testing, and any setting with repeated measurements—treat the ranking rule as a modeling choice that can substantially affect conclusions. By offering a common tensor‑based API for dozens of ranking families, Scorio.jl lowers the barrier to performing systematic sensitivity analyses, reduces custom integration effort, and improves reproducibility.

In conclusion, Scorio.jl delivers a comprehensive, extensible Julia framework for ranking repeated stochastic responses. It unifies direct score‑based, pairwise, psychometric, voting, graph, and listwise methods under a single API, enabling researchers and engineers to experiment with ranking assumptions without rewriting data pipelines. Pilot experiments demonstrate that most methods recover synthetic rankings quickly and remain stable even with few trials, while runtime measurements reveal substantial differences in computational cost across families. The package thus serves as a practical workbench for comparative ranking analysis in any domain where stochastic benchmark data are prevalent.