Learning to Continually Learn via Meta-learning Agentic Memory Designs

The statelessness of foundation models bottlenecks agentic systems’ ability to continually learn, a core capability for long-horizon reasoning and adaptation. To address this limitation, agentic systems commonly incorporate memory modules to retain and reuse past experience, aiming for continual learning during test time. However, most existing memory designs are human-crafted and fixed, which limits their ability to adapt to the diversity and non-stationarity of real-world tasks. In this paper, we introduce ALMA (Automated meta-Learning of Memory designs for Agentic systems), a framework that meta-learns memory designs to replace hand-engineered memory designs, therefore minimizing human effort and enabling agentic systems to be continual learners across diverse domains. Our approach employs a Meta Agent that searches over memory designs expressed as executable code in an open-ended manner, theoretically allowing the discovery of arbitrary memory designs, including database schemas as well as their retrieval and update mechanisms. Extensive experiments across four sequential decision-making domains demonstrate that the learned memory designs enable more effective and efficient learning from experience than state-of-the-art human-crafted memory designs on all benchmarks. When developed and deployed safely, ALMA represents a step toward self-improving AI systems that learn to be adaptive, continual learners.

💡 Research Summary

The paper tackles a fundamental bottleneck in modern agentic systems: the stateless nature of foundation models (FMs) prevents them from accumulating and reusing experience during inference, limiting continual learning and long‑horizon reasoning. Existing solutions add memory modules—token‑level, parametric, or latent—but these designs are handcrafted, fixed, and domain‑specific, requiring extensive human effort to tailor each to a new task distribution.

ALMA (Automated meta‑Learning of Memory designs for Agentic systems) proposes to replace human‑engineered memory with automatically discovered designs. The core idea is to treat memory design as a program synthesis problem: the search space consists of executable Python code, which is theoretically Turing‑complete and can express any conceivable memory architecture, from simple key‑value stores to complex graph‑based schemas with custom retrieval and update logic.

A Meta Agent, powered by a large language model, iteratively explores this space through an open‑ended loop: (1) sample existing designs from an archive together with their evaluation logs; (2) reflect on successes and failures to generate a new design plan; (3) implement the plan by generating and debugging code; (4) evaluate the new memory by plugging it into a fixed base agent and measuring success rates in a “Memory Collection” phase followed by a “Deployment” phase. Errors trigger self‑reflection and up to three refinement attempts before the design is either accepted into the archive or discarded.

To make the search tractable, the authors define a modular abstraction with two universal interfaces—general_update() and general_retrieve()—that can be composed hierarchically. Sub‑modules may maintain their own databases (vector, graph, relational) and pass information downstream, enabling rich, multi‑stage pipelines similar to existing hand‑crafted systems like G‑Memory or ReasoningBank, but without being constrained by any preset template.

Experiments span four sequential decision‑making benchmarks: ALFWorld, TextWorld, Baba‑Is‑You, and MiniHack. Across all domains, the memory designs discovered by ALMA outperform state‑of‑the‑art human‑crafted baselines by an average of 12 percentage points in success rate. The learned designs also scale more efficiently with memory size, delivering higher performance per unit of storage and lower computational overhead. Ablation studies confirm that (i) open‑ended exploration beats greedy selection, (ii) a code‑based search space yields richer designs than a limited parameter space, and (iii) the Meta Agent’s self‑reflection loop is crucial for correcting synthesis errors.

Limitations are acknowledged: the code search space is vast, making early‑stage exploration inefficient; the current work focuses solely on token‑level memory, leaving parametric and latent memory integration for future work; and reliance on large language models incurs substantial compute costs. Potential extensions include smarter sampling strategies (e.g., reinforcement‑learning‑guided proposals), hybrid memory architectures that combine explicit code‑based modules with learned weight‑based encodings, and safety/interpretability frameworks to ensure that automatically generated memory does not introduce undesirable behaviors.

In summary, ALMA demonstrates that memory—one of the most critical components for continual learning—can be meta‑learned automatically. By leveraging open‑ended code synthesis and a reflective Meta Agent, the system discovers domain‑specific memory designs that are both more effective and more efficient than manually engineered counterparts, marking a significant step toward self‑improving AI agents capable of lifelong adaptation.