AGENTIC-SYSTEMSJuly 9, 20261 min read2 views

AgentLens: Production-Assessed Trajectory Reviews for Coding Agent Evaluation

Traditional coding agent benchmarks only evaluate the final pass/fail outcome. AgentLens introduces a comprehensive evaluation framework that assesses the entire interaction trajectory, tool usage, and error recovery of AI agents in production.

Computer Science > Artificial Intelligence

Title:AgentLens: Production-Assessed Trajectory Reviews for Coding Agent Evaluation

View PDFAbstract:We present AgentLens, a production-assessed benchmark for interactive code agents. Most code-agent benchmarks reduce a run to a single bit -- did the task pass? -- but the people who actually use these agents experience the entire trajectory: how the agent follows instructions, uses its tools, verifies its own work, recovers from mistakes, and talks to them along the way. AgentLens evaluates that whole trajectory. It pairs formal verification, where an objective check exists, with LLM-written trajectory reviews and side-by-side comparisons, so that each run yields a readable explanation of why the score is what it is. This makes AgentLens useful for more than ranking models: we use it to diagnose model behavior, compare successive versions of our own agent, and catch product regressions in a nightly evaluation pipeline. We release the benchmark as open source at this https URL.

Current browse context:

Bibliographic and Citation Tools

Code, Data and Media Associated with this Article

Demos

Recommenders and Search Tools

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

Source: arXiv cs.AI Recent

More in this category

agentic-systems

Evaluating SageMath-Augmented LLM Agents for Computational and Experimental Mathematics

A new study evaluates the integration of SageMath, a Computer Algebra System (CAS), into LLM agents to solve research-level mathematical problems. The results show significant performance gains across all models, with GPT-5.5 achieving the highest solve rate of 75.2%, paving the way for automated conjecture discovery.

agentic-systems

Operational Reframing and Approval-Framed Delegation in Multi-Agent LLM Safety

A new study warns that current safety evaluations underestimate the risks of multi-agent LLM systems, showing how harmful prompts can bypass safety filters through operational reframing and approval-framed delegation.

agentic-systems

LLM-powered reasoning in agent-based modeling

Researchers have introduced HALE, a hybrid framework combining agent-based modeling (ABM) with large language models (LLMs) to better predict human decision-making. This innovative approach could significantly improve policy-making and epidemic simulation.

agentic-systems

Measuring Intelligence Beyond Human Scale

As AI capabilities surpass human limits, traditional benchmarks fail. This paper proposes a new paradigm of relative measurement where AI models generate challenges to evaluate each other, creating a scalable, adversarial rating system.

agentic-systems

From Atomic Actions to Standard Operating Procedures: Iterative Tool Optimization for Self-Evolving LLM Agents

Researchers have introduced EvoSOP, a framework that enables LLM agents to synthesize granular atomic actions into reusable Standard Operating Procedures (SOPs). This iterative tool optimization significantly boosts task success rates and reduces reasoning overhead, paving the way for self-evolving AI systems.

agentic-systems

InductWave: Inductive Multi-Hop Logical Query Answering on Knowledge Graphs

Researchers have introduced InductWave, a wavelet-based inductive embedding method designed for multi-hop logical query answering on massive knowledge graphs. By reasoning over unseen entities with significantly fewer computational resources, InductWave outperforms state-of-the-art models while reducing the required message-passing layers.