Hindsight is 20/20: Building Agent Memory that Retains, Recalls, and Reflects

Hindsight Technical Report HINDSIGHT IS 20/20: BUILDING AGENT MEMORY THAT RETAINS, RECALLS, AND REFLECTS Chris Latimer♣, Nicoló Boschi♣, Andrew Neeser♢, Chris Bartholomew♣, Gaurav Srivastava♡, Xuan Wang♡, Naren Ramakrishnan♡ ♣Vectorize.io, USA ♢The Washington Post, USA ♡Virginia Tech, USA ABSTRACT Agent memory has been touted as a dimension of growth for LLM-based appli- cations, enabling agents that can accumulate experience, adapt across sessions, and move beyond single-shot question answering. The current generation of agent memory systems treats memory as an external layer that extracts salient snippets from conversations, stores them in vector or graph-based stores, and retrieves top-k items into the prompt of an otherwise stateless model. While these systems improve personalization and context carry-over, they still blur the line between evidence and inference, struggle to organize information over long horizons, and offer limited support for agents that must explain their reasoning. We present HINDSIGHT, a memory architecture that treats agent memory as a structured, first-class substrate for reasoning by organizing it into four logical networks that distinguish world facts, agent experiences, synthesized entity summaries, and evolving beliefs. This framework supports three core operations—retain, recall, and reflect—that gov- ern how information is added, accessed, and updated. Under this abstraction, a temporal, entity-aware memory layer incrementally turns conversational streams into a structured, queryable memory bank, while a reflection layer reasons over this bank to produce answers and to update information in a traceable way. On key long-horizon conversational memory benchmarks like LongMemEval and Lo- CoMo, Hindsight with an open-source 20B model lifts overall accuracy from 39% to 83.6% over a full-context baseline with the same backbone and outperforms full-context GPT-4o. Scaling the backbone further pushes Hindsight to 91.4% on LongMemEval and up to 89.61% on LoCoMo (vs. 75.78% for the strongest prior open system), consistently outperforming existing memory architectures on multi-session and open-domain questions. 1 INTRODUCTION AI agents are increasingly expected to behave less like stateless question answering systems and more like long-term partners: they are expected to remember past interactions, build up and track knowledge about the world, and maintain stable perspectives over time Packer et al. (2023); Rasmussen et al. (2025). However, the current generation of agent memory systems today are still built around short- context retrieval-augmented generation (RAG) pipelines and generic large language models (LLMs). Such designs treat memory as an external layer that extracts salient snippets from conversations, stores them in vector or graph-based stores, and retrieves top-k items into the prompt of an otherwise stateless model Wu et al. (2024); Maharana et al. (2024). As a result, current approaches to modeling agent memory struggle with three recurring challenges. First, they are unable to preserve and granularly access long-term information across sessions Tavakoli et al. (2025); Ai et al. (2025). Second, AI agents are unable to epistemically distinguish what the agent has observed from what it believes. Finally, such agents are notorious for their inability to exhibit preference consistency, i.e., expressing a stable reasoning style and viewpoint across interactions rather than producing locally plausible but globally inconsistent responses Huang et al. (2025). 1 arXiv:2512.12818v1 [cs.CL] 14 Dec 2025 Hindsight Technical Report Recent work has begun to address these challenges through dedicated memory architectures for agents, e.g., see Zhang et al. (2025b); Wu et al. (2025). Systems like MemGPT Packer et al. (2023) introduce operating system-like memory management, while Zep Rasmussen et al. (2025) proposes temporal knowledge graphs as an internal data structure. Other approaches focus on continual learning Ai et al. (2025), reinforcement-based memory management Yan et al. (2025), or production- ready memory systems Chhikara et al. (2025). While these systems improve personalization and context carry-over, they still blur the line between evidence and inference, can struggle to selectively organize information over long horizons, and offer limited support for agents that must explain why they answered a question a certain way. We present HINDSIGHT, a memory architecture for long-lived AI agents that addresses these chal- lenges by unifying long-term factual recall with preference-conditioned reasoning. Each agent in HINDSIGHT is backed by a structured memory bank that accumulates everything the agent has seen, done, and decided over time, and a reasoning layer that uses this memory to answer questions, execute workflows, form opinions, and update beliefs in a consistent way. Conceptually, HINDSIGHT ties together two components: TEMPR (Temporal Entity Memory Priming Retrieval),

This content is AI-processed based on open access ArXiv data.