DRBENCHER: Can Your Agent Identify the Entity, Retrieve Its Properties and Do the Math?

DRBENCHER is a new benchmark generator designed to evaluate AI agents on tasks requiring both web browsing and multi-step computation, revealing that even top-tier models achieve only 20% accuracy.

Computer Science > Artificial Intelligence

Title:DRBENCHER: Can Your Agent Identify the Entity, Retrieve Its Properties and Do the Math?

Deep research agents increasingly interleave web browsing with multi-step computation, yet existing benchmarks evaluate these capabilities in isolation, creating a blind spot in assessing real-world performance. We introduce DRBENCHER, a synthetic benchmark generator for questions that require both browsing and computation. It enforces four criteria: verifiability (gold answers are computed by executing parameterized code over knowledge-graph values), complexity (multi-hop entity identification, property retrieval, and domain-specific computation), difficulty (a two-stage verification cascade filters out questions solvable by the generating model), and diversity (a greedy max-min embedding filter maximizes coverage). These criteria are realized via a unified answer-first pipeline spanning five domains: biochemistry, financial, geophysical, security, and history. Human evaluation shows 76% validity (84% excluding stale data), with 35% of errors due to outdated knowledge-graph entries, highlighting an inherent limitation of systems that reason over evolving data. Automatic evaluation shows that the strongest frontier model achieves only 20% answer accuracy. Compared to manually constructed benchmarks (BrowseComp+, MATH-500, GPQA), DRBENCHER achieves the highest semantic diversity.

Source: arXiv cs.AI Recent

More in this category

agentic-systems

VAMPS: Visual-Assisted Mathematical Problem Solving Benchmark

Researchers have introduced VAMPS, a new benchmark designed to evaluate how multimodal large language models leverage visualization tools for mathematical reasoning. The study reveals a surprising gap, showing that models still perform better with direct analytical solving than with tool-enabled visual plotting.

agentic-systems

StepPRM-RTL: Stepwise Process-Reward Guided LLM Fine-Tuning for Enhanced RTL Synthesis

Researchers have introduced StepPRM-RTL, a novel framework combining process-reward modeling (PRM) and Monte Carlo Tree Search (MCTS) to enhance LLM-based RTL code generation. The method improves functional correctness and reasoning fidelity by over 10%, setting a new standard for hardware design automation.

NOW LET US Related – The Saturation Trap and the Subjectivity of Intervention Timing: Why Affect-Based Triggers and LLM Judges Fail to Time Interventions on Autonomous Agents

agentic-systems

The Saturation Trap and the Subjectivity of Intervention Timing: Why Affect-Based Triggers and LLM Judges Fail to Time Interventions on Autonomous Agents

A new study reveals critical flaws in timing interventions on autonomous AI agents, showing that both affect-based triggers and LLM judges fail due to the 'saturation trap' and highly subjective human standards.

EXPLORE TOPICS