AGENTIC-SYSTEMSJuly 7, 20261 min read

APeB: Benchmarking Personalization Ability of Large Language Model Agents

A new benchmark called APeB has been introduced to evaluate the personalization capabilities of LLM agents. The study reveals that current models struggle to handle vague user queries and leverage interaction history effectively.

Computer Science > Artificial Intelligence

Title:APeB: Benchmarking Personalization Ability of Large Language Model Agents

View PDF HTML (experimental)Abstract:LLM-powered agents struggle with personalization when users issue raw, underspecified queries. In this setting, agents must infer latent intent, extract preferences from noisy interaction histories, and select among competing alternatives. Existing benchmarks rarely test this capability, as they often rely on user-refined queries or simplified histories. We introduce personalized product search (PPS), a testbed for agentic personalization under raw queries and diverse histories. We construct Agent Personalized Benchmark (APeB) from action logs, pairing underspecified intents with rich histories and user-viewed candidate items. Evaluating state-of-the-art LLMs with multi-step agent workflows, we find that models handle explicit queries well but struggle with early-stage queries requiring intent and preference discovery. Rubric analysis attributes this gap mainly to ineffective history use. A simple history-aware query-refinement pipeline, VQRA, yields consistent gains, highlighting the need for dedicated history-utilization modules in personalized agents.

Bibliographic and Citation Tools

Code, Data and Media Associated with this Article

Demos

Recommenders and Search Tools

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

Source: arXiv cs.AI Recent

Stay in the loop

Get the best AI & tech insights in your inbox — no spam, unsubscribe anytime.

More in this category

agentic-systems

Automated Data Readiness for Scientific AI

Researchers have introduced REDI, an open-source framework that automates the transformation of raw scientific datasets into AI-ready data. Tested on the Frontier supercomputer, REDI addresses the data preparation bottleneck in scientific AI domains.

agentic-systems

ASK in the Dark: Uncertainty-Gated LLM Assistance under Partial Observability

Researchers have introduced ASK+, a novel framework that significantly enhances small language model (SLM) assistance for reinforcement learning agents under partial observability. By providing trajectory-aware context and structured reasoning, ASK+ demonstrates that prompt design and selective gating can outperform sheer model scale.

agentic-systems

When Aggregate Alignment Misleads: Auditing Policy Repair Without Per-State Expert Actions

Evaluating agentic policy repair without per-state expert labels is challenging, as aggregate behavioral alignment metrics can be highly misleading. A new study using a hotel-pricing simulator demonstrates that LLM-based editors can achieve near-benchmark revenue by leveraging indirect diagnostic feedback, highlighting the need for closed-loop evaluation.

agentic-systems

Embodied Operators and Benchmarking: Toward Reusable and Deployable Embodied Intelligence Systems

A new study proposes 'embodied operators' to modularize embodied intelligence systems, enabling reusable and deployable robot pipelines beyond end-to-end models, supported by a multi-dimensional benchmarking framework.

agentic-systems

Evaluating Generative Agents with Actions Grounded in Socially Distributed Task Environments using Incognita

Researchers introduce Incognita, a framework to evaluate generative agents in socially distributed task environments where knowledge is partitioned across multiple entities. Testing shows that while advanced models perform better at gathering hidden information, overall reliability in executing grounded actions remains low.

agentic-systems

Object-Centric Environment Modeling for Agentic Tasks

Researchers propose Object-Centric Environment Modeling (OCM) to help AI agents organize accumulated experiences into an executable programmatic model, addressing the limitations of traditional free-form textual memories.