AutoEnv: Automated Environments for Measuring Cross-Environment Agent Learning

Humans naturally adapt to diverse environments by learning underlying rules across worlds with different dynamics, observations, and reward structures. In contrast, existing agents typically demonstrate improvements via self-evolving within a single domain, implicitly assuming a fixed environment distribution. Cross-environment learning has remained largely unmeasured: there is no standard collection of controllable, heterogeneous environments, nor a unified way to represent how agents learn. We address these gaps in two steps. First, we propose AutoEnv, an automated framework that treats environments as factorizable distributions over transitions, observations, and rewards, enabling low-cost (4.12 USD on average) generation of heterogeneous worlds. Using AutoEnv, we construct AutoEnv-36, a dataset of 36 environments with 358 validated levels, on which seven language models achieve 12-49% normalized reward, demonstrating the challenge of AutoEnv-36. Second, we formalize agent learning as a component-centric process driven by three stages of Selection, Optimization, and Evaluation applied to an improvable agent component. Using this formulation, we design eight learning methods and evaluate them on AutoEnv-36. Empirically, the gain of any single learning method quickly decrease as the number of environments increases, revealing that fixed learning methods do not scale across heterogeneous environments. Environment-adaptive selection of learning methods substantially improves performance but exhibits diminishing returns as the method space expands. These results highlight both the necessity and the current limitations of agent learning for scalable cross-environment generalization, and position AutoEnv and AutoEnv-36 as a testbed for studying cross-environment agent learning. The code is avaiable at https://github.com/FoundationAgents/AutoEnv.

💡 Research Summary

The paper tackles a fundamental gap in modern reinforcement‑learning and large‑language‑model research: while humans effortlessly acquire abstract rules that transfer across worlds with different dynamics, observations, and reward structures, current agents are evaluated almost exclusively within a single, static environment distribution. Consequently, there is no standard suite of heterogeneous, controllable environments, nor a unified formalism for describing how an agent improves its internal components. To close these gaps, the authors introduce two contributions.

First, they present AutoEnv, an automated framework that treats an environment as a factorizable distribution over three independent factors—state transitions, observations, and rewards. By sampling each factor from user‑specified probability distributions, AutoEnv can generate a virtually unlimited variety of worlds at very low cost (averaging $4.12 per environment in their cloud‑based implementation). Using this system they construct AutoEnv‑36, a curated collection of 36 distinct environments containing a total of 358 validated levels. Each level has been manually inspected to ensure a balanced difficulty spectrum and genuine heterogeneity: the underlying physics, sensory modalities, and incentive structures differ markedly from one another.

Second, the authors formalize agent learning as a component‑centric pipeline consisting of three stages: Selection, Optimization, and Evaluation. In the Selection stage an improvable component of the agent (e.g., policy network, value estimator, exploration heuristic) is identified. In the Optimization stage a concrete learning method is applied to improve that component. In the Evaluation stage the resulting agent is run on a target environment and its performance is measured using a normalized reward metric. This abstraction allows the systematic comparison of many learning strategies on the same set of environments.

Guided by this formulation, the paper designs eight learning methods that span meta‑learning, evolutionary search, prompt‑based fine‑tuning, hyper‑parameter optimization, and hybrid combinations. They first benchmark seven large language models on AutoEnv‑36 without any adaptation, observing normalized rewards ranging from 12 % to 49 %—a clear indication that the dataset is challenging for current models. When each of the eight methods is applied individually, performance gains quickly diminish as the number of environments grows, demonstrating that a single fixed learning pipeline does not scale to heterogeneous settings.

To address this limitation, the authors introduce an environment‑adaptive selection mechanism that, for each environment, automatically chooses the most promising learning method from the pool. This meta‑level adaptation yields a substantial overall boost in normalized reward, confirming that method selection matters. However, the benefit plateaus as the method space expands, highlighting a diminishing‑returns phenomenon: the more candidate methods are available, the harder it becomes to reliably pick the best one without incurring additional overhead.

The empirical findings lead to several key insights. (1) Cross‑environment generalization remains an open problem; existing meta‑learning and evolutionary techniques are insufficient when faced with truly diverse dynamics and reward schemas. (2) Dynamic method selection is a powerful lever, but it introduces a new layer of complexity that itself must be learned or optimized. (3) AutoEnv and AutoEnv‑36 provide a cost‑effective, reproducible benchmark for future research on scalable, cross‑environment agent learning.

In the discussion, the authors outline promising research directions: extending the factorization model to capture richer physical and social interactions; developing meta‑optimizers that can learn to select or even synthesize learning methods on the fly; and exploring structured representation learning that mirrors human rule abstraction. By delivering both a practical toolset and a rigorous experimental framework, the paper positions AutoEnv as a foundational testbed for the next generation of agents capable of learning how to learn across a spectrum of worlds.