Agent Observability Powers Agent Evaluation

TL;DR

• You don't know what your agents will do until you actually run them — which means agent observability is different and more important than software observability
• Agents often do complex, open-ended tasks, which means evaluating them is different than evaluating software
• Because traces document where agent behavior emerges, they power evaluation in a multitude of ways

When something goes wrong in traditional software, you know what to do: check the error logs, look at the stack trace, find the line of code that failed. But AI agents have changed what we're debugging. When an agent takes 200 steps over two minutes to complete a task and makes a mistake somewhere along the way, that’s a different type of error. There’s no stack trace - because there’s no code that failed. What failed was the agent’s reasoning.

From debugging code to debugging reasoning

You still write code to define your agent, e.g. which tools exist, what data is available. You write prompts and tool descriptions to guide the agent's behavior, but you won't know how the LLM will interpret these instructions until you run it. The source of truth thus shifts from code to traces that show what the agent actually did.

Agent engineering is an iterative process, and tracing + evaluation are how you close the loop. In this post, we'll explore why agent observability and evaluation are fundamentally different from traditional software, what new primitives and practices you need, and how observability powers evaluation in ways that make them inseparable.

Agent observability ≠ software observability

Pre-LLMs, software was largely deterministic — given the same input, you'd get the same output. Logic was codified. You could read the code and know exactly how the system behaved. When something went wrong, logs pointed you to which service or function failed, then you'd go back to the code to understand why it happened and fix it.

AI agents break the assumptions of determinism and code as a source of truth. As we're moving from traditional software to LLM applications to agents, each step introduces more uncertainty. LLM apps make a single call to an LLM with context, introducing natural language's inherent "fuzziness" but remain constrained to one LLM call.

However, agents call LLMs and tools in a loop until they determine a task is done — and can reason across dozens or hundreds of steps, calling tools, maintaining state, and adapting behavior based on context. When building an agent, you attempt to recommend the application logic in code and prompts. But you don't know what this logic will do until actually running the LLM.

When something goes wrong, you're not finding a single line of code that failed. Instead, you're asking:

• Why did the agent decide to call edit_file instead of read_file at step 23 of 200?
• What context and prompt instructions informed that decision?
• Where in this two-minute, 200-step trajectory did the agent go off track?

Traditional tracing tools can't answer these questions. A 200-step trace is too large for a human to parse, and traditional traces don't capture the reasoning context behind each decision; they only capture which services were called and how long each took.

Agent evaluation ≠ software evaluation

Traditional software testing relies on deterministic assertions: write tests that check output == expected_output, verify they pass, then ship. Online evaluation (A/B tests, product analytics) measures business impact separately. Evaluating agents differ from evaluating software in a few key ways:

1. You're testing reasoning, not code paths

Traditional software has tests at different levels of granularity (unit, integration, e2e), testing deterministic code paths you can read and modify. Agents also need testing at different levels, but you're no longer testing code paths — you're testing reasoning:

• Single-step: Did the agent make the right decision at this moment?
• Full-turn: Did the agent perform well in an end-to-end execution?
• Multi-turn: Did the agent maintain context across a conversation?

2. Production becomes your primary teacher

In traditional software, you can catch most correctness issues with offline tests (unit tests, integration tests, staging). You still test in production through canary deployments and feature flags, but the goal is to catch edge cases and integration issues you missed.

With agents, production plays a different role. Because every natural language input is unique, you can't anticipate how users will phrase requests or what edge cases exist. Production traces reveal failure modes you couldn't have predicted and help you understand what "correct behavior" actually looks like for real user interactions.

This shifts how you think about evaluation: production isn't just where you catch missed bugs. It's where you discover what to test for offline. Production traces become test cases, and your evaluation suite grows continuously from real-world examples, not just engineered scenarios.

The primitives of agent observability

Agent observability uses three core primitives to capture non-deterministic reasoning:

• Runs: A single execution step (one LLM call with its input/output)
• Traces: A complete agent execution showing all runs and their relationships
• Threads: Multi-turn conversations grouping multiple traces over time

Runs: capturing what the LLM did at a single step

A run captures a single execution step. This is most useful for capturing how the LLM behaved at a particular point in time. This captures the complete prompt for an LLM call, including all instructions, tools used, and context.

Traces: capturing trajectories

A trace captures a complete agent execution by linking together all the runs that occurred. A reasoning trace captures all information about what goes into the model at each step, all tool calls, and the nested structure showing how steps relate to each other. Agent traces are massive, sometimes reaching hundreds of megabytes.

Threads: multi-turn conversation context

A single trace captures one agent execution, but agents often operate across sessions. A thread groups multiple agent executions (traces) into a single conversational session, preserving multi-turn context and state evolution. Threads are essential for understanding how agent behavior evolves over time and how context accumulates across interactions.