AGENTIC-SYSTEMSJuly 15, 20261 min read

PM-Bench: Evaluating Prospective Memory in LLM Agents

A new study introduces PM-Bench, a benchmark designed to evaluate the 'prospective memory' of LLM agents. The evaluation reveals that even the most advanced large language models struggle significantly with remembering and executing scheduled tasks.

Computer Science > Artificial Intelligence

Title:PM-Bench: Evaluating Prospective Memory in LLM Agents

View PDF HTML (experimental)Abstract:A significant challenge in agentic AI is prospective memory: the ability to execute an intention at a specific future cue or state while other activities are ongoing. We introduce PM-Bench, a text-based benchmark for measuring prospective memory capabilities in modern LLM agents. Inspired by the Virtual Week paradigm from cognitive science, PM-Bench evaluates how well LLM agents maintain user intentions, execute delayed intentions, and monitor latent environment changes. Over the course of a simulated seven-day week, agents must continue an ongoing activity while deciding whether any deferred task is due. We compare eight state-of-the-art LLMs on PM-Bench under eight different agent configurations. PM-Bench proves challenging across all settings: the best method, a GPT-5.4 agent, reaches only 65.1% F1 score under our evaluation. Furthermore, no single strategy for improving prospective memory dominates across models. We release PM-Bench as a controlled testbed for diagnosing these failures and developing training or inference-time interventions that support reliable prospective behavior.

Bibliographic and Citation Tools

Code, Data and Media Associated with this Article

Demos

Recommenders and Search Tools

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

Source: arXiv cs.AI Recent

More in this category

agentic-systems

LP Mining with LP2Graph: A Use Case for Railway Rescheduling

Researchers introduce LP Mining with LP2Graph, a method to mine and standardize Mixed-Integer Linear Programming (MILP) formulations into a reproducible dataset, establishing a foundation for automated railway rescheduling.

agentic-systems

A Threshold Exceedance Framework for CBRN Uplift Evaluation in Frontier Language Models

Researchers introduce the Threshold Exceedance Criteria (TEC) framework to standardize how we evaluate whether frontier AI models materially increase the risk of non-experts planning CBRN (Chemical, Biological, Radiological, or Nuclear) attacks.

NOW LET US Related – Designing Agent-Ready Websites for AI Web Agents: A Framework for Machine Readability, Actionability, and Decision Reliability

agentic-systems

Designing Agent-Ready Websites for AI Web Agents: A Framework for Machine Readability, Actionability, and Decision Reliability

This paper introduces the 'agent-ready website' framework, designed to optimize e-commerce platforms for AI web agents. Experimental results show that optimizing websites for machine readability and actionability increases AI agent task success rates from 49.3% to 89.3% while reducing step counts and token consumption.

agentic-systems

On-Device Deep Research at 4B: Exposure Bounds Faithfulness, Retrieval Bounds Coverage

A new study explores how to optimize on-device AI research agents running on personal laptops, revealing that context exposure limits citation faithfulness while retrieval quality limits information coverage.

agentic-systems

Good Benchmarks

As AI models advance rapidly, establishing high-quality benchmarks is more critical than ever. A recent paper outlines the core criteria that define a truly effective and practical AI evaluation system.

agentic-systems

How Many Tasks Are Enough for Agent Benchmark Decisions? A Replay Analysis of Public LLM Agent Benchmarks

Evaluating LLM agents is costly, tempting researchers to run only a fraction of benchmark tasks. However, a new replay analysis reveals that the required task fraction varies wildly across benchmarks, warning that partial evaluations can easily lead to incorrect pairwise conclusions.