Active Evaluation of General Agents: Problem Definition and Comparison of Baseline Algorithms

As intelligent agents become more generally-capable, i.e. able to master a wide variety of tasks, the complexity and cost of properly evaluating them rises significantly. Tasks that assess specific capabilities of the agents can be correlated and stochastic, requiring many samples for accurate comparisons, leading to added costs. In this paper, we propose a formal definition and a conceptual framework for active evaluation of agents across multiple tasks, which assesses the performance of ranking algorithms as a function of number of evaluation data samples. Rather than curating, filtering, or compressing existing data sets as a preprocessing step, we propose an online framing: on every iteration, the ranking algorithm chooses the task and agents to sample scores from. Then, evaluation algorithms report a ranking of agents on each iteration and their performance is assessed with respect to the ground truth ranking over time. Several baselines are compared under different experimental contexts, with synthetic generated data and simulated online access to real evaluation data from Atari game-playing agents. We find that the classical Elo rating system – while it suffers from well-known failure modes, in theory – is a consistently reliable choice for efficient reduction of ranking error in practice. A recently-proposed method, Soft Condorcet Optimization, shows comparable performance to Elo on synthetic data and significantly outperforms Elo on real Atari agent evaluation. When task variation from the ground truth is high, selecting tasks based on proportional representation leads to higher rate of ranking error reduction.

💡 Research Summary

The paper addresses the growing challenge of evaluating increasingly general AI agents that must be assessed across many tasks. Traditional multi‑task benchmarks rely on static, pre‑collected datasets, which become costly when tasks are noisy, correlated, or when each evaluation (e.g., a game play or a language‑model prompt) is expensive. To mitigate this, the authors introduce a formal “active evaluation” framework in which a ranking algorithm actively selects, at each iteration, a task and a pair of agents whose scores it will observe. The observed pairwise scores are then used to update the algorithm’s current ranking, and the ranking error is measured against a ground‑truth ordering.

The core performance metric is the Average Generalized Top‑k Ranking Error (A GRE). A GRE combines two components: (i) Top‑k Identification Error (IDE), which penalizes mis‑identifying the set of the best k agents, and (ii) normalized Kendall‑tau distance (Kₙ), which measures the overall ordering error. A weighting factor α(k) = (m‑k)/(m‑1) balances the two, giving more weight to IDE when k is small and to Kₙ when k is large. By averaging GRE over all time steps up to horizon T, A GRE captures the cumulative efficiency of a ranking algorithm.

The authors adapt several existing ranking methods to the online setting:

1. Elo – a classic logistic rating system. They implement an active version using maximum‑entropy sampling to decide which pairwise match to query, effectively performing online logistic regression.
2. Voting‑as‑Evaluation (VasE) – treats each task as a vote and applies social‑choice rules (Borda, Condorcet, etc.). They combine VasE with adversarial bandit algorithms (Exp3) to choose tasks adaptively.
3. Nash Averaging – constructs a meta‑game where agents play each other across tasks, then extracts ratings from a Nash equilibrium, preserving invariance to cloned agents.
4. Soft Condorcet Optimization (SCO) – a gradient‑descent method that minimizes a smooth approximation of Kendall‑tau loss. SCO guarantees that any Condorcet winner (an agent beating every other agent in pairwise majority) will be placed at the top of the ranking.

To evaluate these methods, the paper uses two synthetic data generators and a simulated online version of the Arcade Learning Environment (ALE). The synthetic generators are:

• Mallows model – draws task‑specific rankings from a central ground‑truth ranking with dispersion parameter φ. Small φ yields highly correlated tasks; φ = 1 yields uniform randomness.
• Task‑Variation model – assigns each task its own noise level, creating realistic heterogeneity across tasks.

Both synthetic setups provide a known ground‑truth ordering, enabling direct computation of A GRE. The real‑world experiment accesses scores of 57 Atari games for several deep‑RL agents, a human baseline, and a random policy, mimicking an online oracle that returns a pairwise score when queried.

Key findings:

• Elo performs robustly across most settings. Even though Elo is theoretically vulnerable to non‑transitive cycles and lacks explicit task modeling, its simplicity and well‑studied update rule lead to rapid reduction of ranking error, especially when tasks are highly correlated (low φ).
• SCO matches Elo on synthetic data but outperforms it on the Atari benchmark, achieving 10–15 % lower A GRE when task variation is high. Its smooth loss and Condorcet guarantee make it more resilient to noisy, heterogeneous tasks.
• VasE and Nash Averaging show competitive performance in limited regimes (few tasks, many agents) but generally lag behind Elo and SCO in overall efficiency.
• Regarding task‑selection strategies, the authors compare random sampling, Upper‑Confidence‑Bound (UCB) style uncertainty sampling, and a “Proportional Representation” heuristic that samples tasks proportionally to their current uncertainty. The proportional approach consistently yields faster convergence, confirming that balanced coverage of diverse tasks is crucial when task‑to‑ground‑truth variance is large.

The paper also contrasts active evaluation with conventional “pre‑process‑then‑evaluate” pipelines. In the latter, the dataset is fixed before ranking, limiting the ability to allocate evaluation budget adaptively. Active evaluation treats the selection of which task‑agent pair to query as part of the optimization problem, thereby achieving substantial cost savings while preserving or improving ranking accuracy.

Finally, the authors outline future directions: developing ranking rules that are robust to non‑transitive cycles and cloned agents, integrating Bayesian uncertainty estimates for more principled active sampling, and scaling the framework to massive language‑model evaluations where each query can cost hundreds of dollars.

In summary, this work provides the first systematic formulation of active evaluation for general AI agents, introduces a suite of adapted ranking algorithms, and demonstrates—through both synthetic and realistic Atari experiments—that a carefully designed active sampling strategy combined with Soft Condorcet Optimization can substantially reduce evaluation cost while delivering accurate agent rankings.