MemTrace: Probing What Final Accuracy Misses in Long-Term Memory

While LLM agents increasingly maintain long-term memory, traditional evaluation methods fail to show how facts behave under changing conditions. The new MemTrace benchmark reveals that the primary bottleneck in AI memory is evidence utilization, not retrieval.

Computer Science > Artificial Intelligence

Title:MemTrace: Probing What Final Accuracy Misses in Long-Term Memory

View PDF HTML (experimental)Abstract:LLM agents increasingly maintain long-term memory of user facts across sessions. Yet such memory is usually evaluated by aggregating accuracy over question rows or episodes. Because this approach scores question rows independently, even when several questions probe the same fact, it cannot show how that fact behaves as conditions change. We introduce MemTrace, a benchmark whose unit of measurement is the knowledge point: a single typed fact about the user, rather than an individual question. MemTrace probes each fact along three controlled dimensions: memory age, defined by how many sessions ago the fact appeared in the history; question type, covering current state, earlier state, and trajectory of change; and evidence condition, covering present, missing, and contradicted-by-false-premise settings. Evaluating 13 memory-system configurations across four paradigms, we find that similar pooled accuracy hides different failures: recovering a fact's current and earlier states does not imply tracking how it changed, and safe abstention does not imply correcting a false premise. The dominant bottleneck is evidence use, not retrieval: when systems fail, the evidence was retrievable 10 times more often than it was missing. These results suggest that improving long-term memory requires better use of reachable evidence, not simply more storage or retrieval.

Bibliographic and Citation Tools

Code, Data and Media Associated with this Article

Demos

Recommenders and Search Tools

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

Source: arXiv cs.AI Recent

More in this category

agentic-systems

Nothing from Something: Can a Language Model Discover 0?

A new study investigates whether AI language models can independently discover the mathematical concept of 'zero'. The findings reveal that while models cannot generalize this concept out-of-the-box, language pretraining reduces the required training examples by 50%.

agentic-systems

SEAGym: An Evaluation Environment for Self-Evolving LLM Agents

Researchers have introduced SEAGym, a new evaluation environment designed to accurately measure the self-evolution of LLM agents. This tool addresses the limitations of traditional evaluation methods, which often overlook overfitting or performance degradation.

agentic-systems

Closing the Feedback Loop: From Experience Extraction to Insight Governance in Verbal Reinforcement Learning

Researchers propose a three-layer architecture (rules, evidence, skills) to close the feedback loop in verbal reinforcement learning, solving the retention-forgetting dilemma for LLM agents in non-stationary environments.

agentic-systems

MapSatisfyBench: Benchmarking Satisfaction-Aware Map Agents through Behavior-Grounded Implicit Decision Factors

Researchers have introduced MapSatisfyBench, a new benchmark for evaluating LLM-based map agents. It shifts the evaluation focus from simple task completion to satisfying implicit user needs and optimizing real-world user satisfaction.

agentic-systems

Surrogate Assisted Pedestrian Protection Design via a Foundation Model Orchestrated Workflow

Researchers have developed the first foundation model-orchestrated workflow for crash safety design, reducing evaluation times from hours of conventional CAE simulations to mere seconds.

agentic-systems

SkillChain-Gym: A Benchmark for Reskilling-Aware Production-Inventory Control under Disruptions

SkillChain-Gym is a new benchmark for production-inventory control that integrates workforce reskilling and training dynamics under disruptions. It evaluates how training policies compete with production capacity and helps decision-makers balance operational efficiency and workforce resilience.

EXPLORE TOPICS