MemAdapter: Fast Alignment across Agent Memory Paradigms via Generative Subgraph Retrieval

Memory mechanism is a core component of LLM-based agents, enabling reasoning and knowledge discovery over long-horizon contexts. Existing agent memory systems are typically designed within isolated paradigms (e.g., explicit, parametric, or latent memory) with tightly coupled retrieval methods that hinder cross-paradigm generalization and fusion. In this work, we take a first step toward unifying heterogeneous memory paradigms within a single memory system. We propose MemAdapter, a memory retrieval framework that enables fast alignment across agent memory paradigms. MemAdapter adopts a two-stage training strategy: (1) training a generative subgraph retriever from the unified memory space, and (2) adapting the retriever to unseen memory paradigms by training a lightweight alignment module through contrastive learning. This design improves the flexibility for memory retrieval and substantially reduces alignment cost across paradigms. Comprehensive experiments on three public evaluation benchmarks demonstrate that the generative subgraph retriever consistently outperforms five strong agent memory systems across three memory paradigms and agent model scales. Notably, MemAdapter completes cross-paradigm alignment within 13 minutes on a single GPU, achieving superior performance over original memory retrievers with less than 5% of training compute. Furthermore, MemAdapter enables effective zero-shot fusion across memory paradigms, highlighting its potential as a plug-and-play solution for agent memory systems.

💡 Research Summary

MemAdapter addresses a fundamental bottleneck in large‑language‑model (LLM)‑based agents: the fragmentation of memory mechanisms into three largely isolated paradigms—explicit (graph‑ or table‑structured external storage), parametric (information embedded in model weights or lightweight adapters such as LoRA), and latent (dynamic key‑value caches or learned embedding spaces). Existing systems tightly couple each paradigm with its own retrieval method, making cross‑paradigm alignment, fusion, and rapid adaptation costly or impossible without extensive architectural redesign and retraining.

The paper proposes a unified framework that decouples retrieval from any specific memory paradigm. Central to MemAdapter is a unified memory space (S) that serves as a common embedding arena for all memory states, regardless of their original format. The system is trained in two stages.

Stage 1 – Generative Subgraph Retrieval via Model Distillation
A strong teacher model is prompted to construct a full memory graph G★ from the underlying data. This graph is processed by a conventional graph‑building pipeline (MemBuild) to obtain an explicit memory state M(a). An “anchored alignment module” Align(a) projects M(a) into the unified space, yielding a representation h(a). The student retriever is then trained to autoregressively generate a subgraph r conditioned on the query q and h(a). Supervision comes from a token‑level KL‑divergence loss that forces the student’s token distribution to match the teacher’s distribution over a serialized version of the ground‑truth subgraph r★ (a subgraph of G★). This distillation step equips the retriever with the ability to synthesize structured evidence without relying on hand‑crafted indexes.

Stage 2 – Cross‑Paradigm Alignment via Contrastive Learning
For a new target paradigm t (e.g., parametric or latent), its memory states M(t) are built with the same MemBuild routine and projected into S by a lightweight alignment module Align(t). Align(a) is frozen; Align(t) is trained on a modest set of 2,500 demonstrations using an InfoNCE‑style loss that maximizes cosine similarity between paired anchored and target representations while pushing apart randomly sampled negatives. This contrastive alignment aligns heterogeneous memory embeddings to the same unified space in under 13 minutes on a single GPU, requiring less than 5 % of the compute needed to fine‑tune the original memory retrievers.

Plug‑and‑Play Retrieval and Fusion
Once Align(t) is learned, any memory state from paradigm t can be instantly transformed into S and fed to the same generative subgraph retriever. Switching paradigms merely involves swapping the alignment module, enabling truly plug‑and‑play retrieval. Moreover, multiple paradigms can be fused by projecting each into S, aggregating the resulting vectors, and letting the retriever generate a subgraph that draws evidence from all sources. The authors demonstrate zero‑shot fusion: even without additional training, combining explicit, parametric, and latent memories yields higher downstream accuracy than any single‑paradigm system.

Empirical Evaluation
The authors evaluate MemAdapter on three public, memory‑intensive question‑answering benchmarks (including long‑context QA, knowledge‑intensive QA, and multi‑hop reasoning) using LLMs ranging from 1.5 B to 7 B parameters. Compared against five strong baselines covering each paradigm (graph‑based retrieval, LoRA adapters, KV‑cache methods, etc.), MemAdapter consistently achieves superior accuracy and F1 scores. Notably, the cross‑paradigm alignment completes in 13 minutes on a single GPU, and the total training budget is under 5 % of that required to fine‑tune the original retrievers.

Contributions and Limitations

1. Introduction of a paradigm‑agnostic generative subgraph retrieval framework.
2. A two‑stage training pipeline that combines model distillation with lightweight contrastive alignment.
3. Demonstration of fast, plug‑and‑play adaptation and zero‑shot fusion across heterogeneous memory sources.

Limitations include the focus on text‑based QA tasks; the approach has not yet been validated on multimodal or sequential decision‑making domains (e.g., robotics, planning). Additionally, the alignment modules are trained separately per paradigm, leaving open the question of joint multi‑paradigm alignment or continual adaptation.

Future Directions
Potential extensions involve (i) aligning multimodal memory (image‑text, audio) to the unified space, (ii) meta‑learning alignment modules for continual, on‑the‑fly adaptation, (iii) scaling to multi‑agent environments where real‑time memory synchronization is required, and (iv) investigating regularization techniques to mitigate alignment bias.

In summary, MemAdapter offers a practical first step toward unified, efficient, and extensible memory retrieval for LLM agents, promising to simplify system design and unlock richer cross‑paradigm knowledge integration.