AgentLongBench: A Controllable Long Benchmark For Long-Contexts Agents via Environment Rollouts

The evolution of Large Language Models (LLMs) into autonomous agents necessitates the management of extensive, dynamic contexts. Current benchmarks, however, remain largely static, relying on passive retrieval tasks that fail to simulate the complexities of agent-environment interaction, such as non-linear reasoning and iterative feedback. To address this, we introduce \textbf{AgentLongBench}, which evaluates agents through simulated environment rollouts based on Lateral Thinking Puzzles. This framework generates rigorous interaction trajectories across knowledge-intensive and knowledge-free scenarios. Experiments with state-of-the-art models and memory systems (32K to 4M tokens) expose a critical weakness: while adept at static retrieval, agents struggle with the dynamic information synthesis essential for workflows. Our analysis indicates that this degradation is driven by the minimum number of tokens required to resolve a query. This factor explains why the high information density inherent in massive tool responses poses a significantly greater challenge than the memory fragmentation typical of long-turn dialogues.

💡 Research Summary

AgentLongBench introduces a controllable, long‑context benchmark designed to evaluate autonomous agents through simulated environment rollouts rather than static document retrieval. The authors build on Lateral Thinking Puzzle environments, where an LLM‑agent interacts with deterministic oracles and auxiliary tools, generating iterative logs of tool calls, tool responses, guesses, and environment feedback. By automatically rolling out these interactions, the benchmark creates coherent, causally‑linked contexts ranging from 32 K to 4 M tokens.

Two orthogonal dimensions are defined: (1) Knowledge‑Intensive, using the Pokémon dataset to trigger parametric knowledge, and (2) Knowledge‑Free, where all entity and attribute names are replaced with abstract tokens to eliminate semantic cues. For each setting, two response formats are employed: Concise‑Response (short tool outputs, many interaction turns) and Verbose‑Response (dense tool outputs, fewer turns). This design isolates the effects of temporal span versus information density on agent performance.

Eight tasks are organized into a taxonomy of 32 question types covering three evaluation stages: QA on Tool Response (testing parsing of machine‑generated logs), QA on Environment Response (testing state tracking across turns), and Final Guess (requiring global logical intersection). Tasks such as FindDuplicates, WeightedSummation, and FindTargetOffsets probe local retrieval, arithmetic precision, and strict positional accuracy respectively.

The experimental suite evaluates a broad set of state‑of‑the‑art LLMs (GPT‑4.1, Gemini‑2.5, Claude‑Sonnet‑4.5, Grok‑4.1) and open‑source long‑context models (DeepSeek‑V3.2, Qwen series, GLM‑4). In addition, external memory mechanisms (RAG, A‑Mem, Mem0, MemoryOS) are benchmarked using Qwen3‑30B‑A3B‑Instruct as a unified backbone.

Key findings:

1. All models excel at short contexts (≤256 K tokens) with accuracies often above 80 %, but performance collapses sharply beyond 1 M tokens, dropping below 30 % for many tasks.
2. Verbose‑Response, which presents high‑density tool logs, is especially challenging; tasks demanding exact positional offsets approach zero accuracy, highlighting the difficulty of parsing large structured outputs.
3. External memory augmentation does not consistently improve results; MemoryOS shows a modest advantage at 32 K tokens but quickly degrades as context grows, while RAG and other augmentations remain flat or worse than the vanilla model. The authors attribute this to a mismatch between generic retrieval pipelines and the need to preserve every constraint as a logical premise.
4. The notion of “minimum token requirement” emerges as a strong predictor of degradation: when the number of tokens necessary to resolve a query exceeds the model’s effective window, accuracy falls dramatically. This effect is more pronounced for dense tool responses than for fragmented long‑turn dialogues.

The paper argues that static reading‑comprehension benchmarks fail to capture the core challenges of autonomous agents, namely dynamic information synthesis, iterative feedback handling, and high‑density log processing. AgentLongBench provides a scalable, deterministic, and controllable framework for diagnosing these failure modes. Limitations include reliance on a single domain (Pokémon) for the knowledge‑intensive setting and a focus on search‑type tools; future work could incorporate richer tool families (code execution, simulation) and broader domains to test generalization.

Overall, AgentLongBench reveals a critical gap: current LLMs, even with extended context windows, are not yet equipped for the non‑linear, constraint‑preserving reasoning required in real‑world agentic workflows. Advancing toward robust autonomous agents will likely require new memory architectures that retain every logical constraint, token‑level parsers capable of handling dense tool outputs, and benchmarks that reflect the full spectrum of dynamic, multi‑turn interactions.