OdysseyArena: Benchmarking Large Language Models For Long-Horizon, Active and Inductive Interactions

The rapid advancement of Large Language Models (LLMs) has catalyzed the development of autonomous agents capable of navigating complex environments. However, existing evaluations primarily adopt a deductive paradigm, where agents execute tasks based on explicitly provided rules and static goals, often within limited planning horizons. Crucially, this neglects the inductive necessity for agents to discover latent transition laws from experience autonomously, which is the cornerstone for enabling agentic foresight and sustaining strategic coherence. To bridge this gap, we introduce OdysseyArena, which re-centers agent evaluation on long-horizon, active, and inductive interactions. We formalize and instantiate four primitives, translating abstract transition dynamics into concrete interactive environments. Building upon this, we establish OdysseyArena-Lite for standardized benchmarking, providing a set of 120 tasks to measure an agent’s inductive efficiency and long-horizon discovery. Pushing further, we introduce OdysseyArena-Challenge to stress-test agent stability across extreme interaction horizons (e.g., > 200 steps). Extensive experiments on 15+ leading LLMs reveal that even frontier models exhibit a deficiency in inductive scenarios, identifying a critical bottleneck in the pursuit of autonomous discovery in complex environments. Our code and data are available at https://github.com/xufangzhi/Odyssey-Arena

💡 Research Summary

OdysseyArena is a newly proposed benchmark suite that shifts the evaluation of large‑language‑model (LLM) based autonomous agents from a deductive, rule‑following paradigm to one that emphasizes long‑horizon, active, and inductive interaction. The authors argue that existing benchmarks largely provide agents with explicit rules, static goals, and short interaction horizons (typically < 50 steps), thereby failing to test an agent’s ability to discover latent transition dynamics from experience—a capability they deem essential for true foresight and strategic coherence.

The paper formalizes an interactive environment as a generative state‑transition function T(sₜ, aₜ) → (sₜ₊₁, rₜ), where the hidden function T encodes the world’s regularities. To systematically probe different kinds of latent structure, T is decomposed into four orthogonal primitives: (1) discrete symbolic rules (Boolean logic over bits), (2) continuous stochastic dynamics (continuous state updates with noise), (3) periodic temporal patterns (cyclic regularities), and (4) relational graph structures (non‑local interactions among entities). Each primitive is instantiated as a lightweight, fully observable‑surface environment that nevertheless hides the underlying dynamics.

The four concrete environments are:

1. Turn On Lights – Implements discrete symbolic rules. A network of N lights is governed by a hidden Boolean circuit. Toggling a single light can cause deterministic cascades across the network. The agent observes the on/off status of all lights after each action but must infer the hidden logical dependencies to achieve the goal (all lights on) within a limited budget.

2. AI Trading – Realizes continuous stochastic dynamics. Multi‑asset market prices evolve according to an unknown function f(s, a) plus stochastic noise. The agent receives price vectors and must learn to predict the dynamics, then devise a buying/selling policy that maximizes profit while managing risk.

3. Energy Dispatch – Captures periodic temporal patterns. Several energy sources (thermal, wind, solar, battery) have production/consumption efficiencies that repeat with a hidden period P. The task is multi‑objective: meet demand, minimize cost, and respect carbon constraints over many days, requiring the agent to discover and exploit the periodicity.

4. Repo System – Embodies relational graph structures. A virtual software repository contains packages with versioned dependencies forming a graph. Installing a package may trigger dependency resolution across the graph. The agent must infer the graph topology and resolve conflicts to achieve a successful installation sequence.

Two benchmark tiers are released:

• OdysseyArena‑Lite – 120 curated tasks spanning the four environments, with typical horizons of 50–100 steps. Designed for rapid iteration, high‑throughput evaluation, and reproducibility. All tasks are API‑driven, allowing easy integration with any LLM‑agent framework.

• OdysseyArena‑Challenge – Stress‑test suite with > 200 steps per episode (some exceeding 1,000 steps) and 10 tasks per environment. This tier probes the limits of memory retention, error recovery, and strategic stability over extremely long interaction sequences.

The authors evaluate 15+ state‑of‑the‑art LLMs, including commercial models (e.g., Gemini 3 Pro Preview, GPT‑4‑Turbo) and a range of open‑source models of varying scales. Results show that even the strongest commercial model attains only a 44.17 % success rate on the Lite suite, far below human performance (≈ 90 %). Open‑source models lag further behind. Detailed analysis reveals systematic failure modes: (i) poor exploration strategies that prevent sufficient data collection for rule induction, (ii) inability to maintain coherent internal representations over long horizons, leading to error accumulation, and (iii) specific difficulty with continuous stochastic dynamics and relational graph environments where the hidden function is highly non‑local.

Key contributions highlighted are: (1) redefining agent evaluation around autonomous discovery of world dynamics, (2) providing a scalable, lightweight set of environments that isolate four fundamental transition primitives, and (3) delivering a comprehensive empirical study that quantifies the inductive bottleneck across leading LLMs. The paper suggests future research directions such as meta‑learning world models, integrating external memory or continual‑learning modules, and leveraging human‑in‑the‑loop feedback to improve exploratory policies.

In sum, OdysseyArena offers the most comprehensive, systematic, and challenging testbed to date for assessing whether LLM‑based agents can truly learn the hidden rules of an environment through long‑term, active interaction—a prerequisite for building genuinely autonomous AI systems.