Evaluating AI Agents: Testing a System That Never Runs the Same Way Twice

Ask an engineer how they know their service works and they'll point at a test suite: unit tests on the parts, integration tests on the seams, a CI gate that goes red when a change breaks something. Ask how they know their agent works and the honest answer is often "we tried it a few times and it looked good." That gap is the single biggest reason agents that demo beautifully fall over in production. The demo is a pass@1 on a friendly input; production is thousands of runs on inputs you never anticipated, over a system that is allowed to be non-deterministic by design. Evaluation is how you close that gap, and it's less a machine-learning problem than a testing problem wearing unfamiliar clothes.

The framing we keep coming back to is that an AI agent is a backend microservice with an inference layer, and in that mapping evals are the test suite. But it's a test suite for code that can return a different answer each time it runs, take a different path to the same answer, and be right for the wrong reasons. Everything hard about agent evaluation follows from that one property, so it's worth being precise about what you measure, how you grade it, and where it lives in your pipeline.

Why Agent Evaluation Is Genuinely Harder

Traditional software is deterministic: same input, same output, and a test is a simple equality check. Agents break all three assumptions at once, and each break is a distinct source of pain.

• The output is non-deterministic. Run the same prompt twice and you can get two different answers, both acceptable. An exact-match assertion is useless; "correct" is now a fuzzy region, not a single string.
• The work is multi-step. A real agentic run chains planning, tool calls, retrieval, retries, and sub-agent handoffs across a long trajectory. Failure can hide in any step, and a single top-line "did it work" hides where it broke.
• There is no canonical path. Two agents can reach the right answer by completely different routes, both valid. Grade the path too strictly and you fail correct runs for not matching your idea of how they should have gotten there.
• Right answers can come from wrong reasoning. An agent can guess, or reach the correct end state through an unsafe or nonsensical path (a "corrupt success"). Outcome-only grading rubber-stamps it.

So agent evaluation is inherently multi-dimensional. The eval stacks that actually hold up in production measure four things at once, and confusing them is where most teams go wrong:

An agent can nail the outcome and still be a disaster on the other three: it called an expensive tool five times when once would do, or wandered through six reasoning steps to answer a one-step question. Conversely it can execute a flawless-looking trajectory and still miss the goal. You need both the destination and the route in view, which is the first fork every eval design has to navigate.

Outcome vs. Trajectory: Grade the Result, Watch the Path

The instinct many teams have is to write down the "correct" sequence of steps and check the agent against it. This feels rigorous and is usually a trap. Pin the trajectory too tightly and you punish the agent for finding a better path than the one you imagined, and your eval breaks every time you change a prompt or swap a model, even when behavior improved. The stronger default, and the one Anthropic's own guidance lands on, is to grade what the agent produced, not the path it took. Score the outcome against the goal, and keep the trajectory for diagnosis, not as the pass/fail gate.

That doesn't mean trajectory is decorative. It's where you answer the questions outcome grading can't: was that success earned or lucky, did the agent call a tool it had no business calling, did it leak a secret or take an irreversible action on the way to a correct result. The clean division of labor: outcome is the gate, trajectory is the debugger. When a run fails, the trajectory tells you whether the agent retrieved the wrong context or reasoned wrong over the right context, the exact same distinction you'd draw between a bad database query and bad business logic in a service. Without it you're debugging a probabilistic system blind.

A useful rule: assert loosely on the path, strictly on the outcome, and reserve hard trajectory assertions for the few steps that are about safety, not style. "The agent must not call refund() without a confirmed order ID" is a real invariant worth failing on. "The agent must call the search tool before the summarize tool" usually isn't; it's you overfitting to one solution.

Capability Is Not Reliability: pass@k vs. pass^k

Here is the metric distinction that separates teams who understand agent reliability from teams who are about to be surprised in production. Because runs are non-deterministic, a single success proves almost nothing. The two ways to summarize many runs pull in opposite directions:

• pass@k — the probability that at least one of k attempts succeeds. It rises toward 100% as you allow more tries. This measures capability: can the agent do this at all?
• pass^k — the probability that all k attempts succeed. It falls as k grows, because pass^k = p^k decays exponentially. This measures consistency: can the agent do this every single time?

At k = 1 they're identical. By k = 10 they've diverged violently: pass@k marches toward 100% while pass^k collapses. The number that stings: an agent with a 90% per-run success rate has a pass^8 of just 0.98 ≈ 43%. Nine-in-ten sounds like a shippable agent; it means that across eight interactions, a customer has better-than-even odds of hitting a failure. The τ-bench work that introduced pass^k found frontier models under 50% at pass^1 and below 25% at pass^8 on retail customer-service tasks. Capability was fine; consistency was the wall.

This is why capability benchmarks alone will mislead you. A leaderboard score is a pass@1 (or worse, a best-of-n), and a customer-facing agent lives and dies on pass^k. The practical takeaway is to run every eval case multiple times and report the distribution, not a single sample. If your eval harness runs each task once, it isn't measuring the property that actually determines whether users trust the agent. Consistency is a first-class axis, and it's the one that quietly decays when you "just change the prompt a little."

How You Actually Grade: Rules, Then Judges

Deciding what to measure is half the job; the other half is the grader that turns a messy natural-language run into a score. There are two families, and mature stacks use both.

Deterministic / rule-based graders come first because they're cheap, fast, and don't lie. Exact match on a structured field, a regex, a JSON-schema check on a tool call, an assertion that a required API was invoked with the right arguments, a check that a forbidden action never fired. Anywhere the correctness criterion is crisp, use code, not a model. It's free, instant, and perfectly repeatable, and it's the backbone of your tool-use and safety checks.

LLM-as-a-judge handles everything rules can't express: is this answer faithful to the retrieved context, is the tone appropriate, did it actually resolve the user's issue, is this summary complete. You hand a second model the input, the agent's output, and a rubric, and ask it to score. It's the only thing that scales open-ended quality grading, and it's the same mechanism we described for output guardrails and semantic-cache verification, just pointed at evaluation instead of runtime. But a judge is itself a non-deterministic system, and it has documented failure modes you have to engineer around:

• Position bias — when comparing two outputs, judges systematically favor whichever came first. Randomize order, or grade absolutely rather than pairwise.
• Verbosity bias — longer answers get scored higher regardless of quality. Control for length in the rubric.
• Self-enhancement bias — a judge favors outputs from its own model family. Use a different model to judge than to generate where you can.
• Surface manipulation — judges can be fooled by confident phrasing over correct content. Anchor the rubric to verifiable facts, not vibes.

The practices that make judges trustworthy are boring and non-negotiable. Prefer binary pass/fail over 1–10 scores; binary judgments are far more stable and reproducible, and a "7" means nothing consistent across runs. Give one judge one dimension rather than asking a single call to rate faithfulness, tone, and completeness at once; isolated judges are sharper and their disagreements are legible. Provide an escape clause ("return Unknown when uncertain") so the judge abstains instead of guessing. And above all, calibrate the judge against human labels: hand-grade a sample, measure agreement, and keep re-checking, because an uncalibrated judge is just a second unverified model you've chosen to trust. A judge you haven't measured is not an evaluation; it's a vibe with a number attached.

Benchmarks Tell You About the Model, Not Your Agent

It's tempting to lean on the public benchmarks, and they're genuinely useful for what they measure. SWE-bench Verified tests agents on real GitHub issues and is the standard for coding ability; τ-bench probes tool-and-policy dialogue in simulated retail and airline support; GAIA measures multi-step assistant tasks; WebArena and OSWorld cover browser and computer use. They're excellent for picking a base model and tracking the frontier. As of mid-2026 the top of SWE-bench Verified sits near 88%, which tells you the models are strong.

What they can't tell you is whether your agent works. Your agent has your tools, your prompts, your data, your policies, and your users' weird phrasings, and none of that is in a public benchmark. A model that tops every leaderboard can still be useless on your task because your retrieval is misconfigured or your tool descriptions are ambiguous. So benchmarks inform model selection; they never substitute for a domain eval set built from your own traffic. And the good news is you don't need thousands of cases to start. The most effective starting point is 20–50 tasks drawn from real failures, actual runs where the agent got it wrong, because early on the effect sizes are large and a handful of well-chosen cases catches most regressions. You grow the set as the agent matures and the easy failures get squeezed out.

Eval-Driven Development: From One-Off Checks to a Regression Gate

The teams who get the most out of evals treat them exactly like unit tests: owned by the people closest to the product, run on every commit, and updated when behavior drifts. This is eval-driven development, and the discipline is the same one that makes ordinary software trustworthy: nothing probabilistic ships without an automated proof it meets spec. In practice the layers mirror a normal test pyramid.

The mechanism that makes this a flywheel rather than a chore is graduation. You write a hard case to answer "can the agent do this at all?" (a capability eval). Once the agent reliably passes it, the question flips to "can it still do this?" and the case graduates into the regression suite, where it runs forever to catch drift. What was once the frontier becomes the baseline you protect. Do this consistently and your eval set becomes a durable record of every failure you've ever fixed, and a regression caught in CI costs minutes instead of a production incident. This is also why the eval loop we described for tuning a semantic cache threshold isn't a special case; it's this same regression discipline applied to one high-risk knob.

The final layer earns the most and is skipped the most: online evaluation. Your offline set, however good, is a fixed snapshot; real users are an endless stream of inputs you didn't imagine. So you instrument production. This is where the microservice framing pays off literally: agents emit OpenTelemetry traces like any other distributed system, with spans for each LLM call, tool execution, and sub-agent handoff in a parent-child tree. On top of that trace data you run a sampled judge, grading a small percentage of live runs for faithfulness, policy compliance, and resolution, and you feed the failures straight back into the eval set. That closes the loop: production surfaces new failure modes, they become eval cases, the eval cases become regression gates, and the agent stops repeating its mistakes.

Think of it as a Swiss-cheese model. No single eval layer catches everything: rules miss nuance, judges have biases, offline sets miss the long tail, online sampling misses what it doesn't sample. Stack them and the holes stop lining up. A failure that slips past the unit eval gets caught by the trajectory eval, and what slips past both gets caught by online monitoring and turned into next week's regression test.

What Actually Bites in Production

The concepts are clean; the pain lives in the details. A field checklist:

• Run every case more than once. A single sample measures capability and hides the consistency problem that determines whether users trust the agent. If you report one number per case, report pass^k, not pass@1.
• Separate retrieval failures from reasoning failures. When an answer is wrong, you must know whether the agent got bad context or reasoned badly over good context. Without that split you'll tune the wrong half of the system.
• Never let an uncalibrated judge gate anything. An LLM judge you haven't measured against human labels is an unverified model grading another unverified model. Calibrate first, then trust, then keep re-checking.
• Assert on invariants, not on style. Hard-code the safety rules (no refund without an order ID) and grade everything else on outcome. Over-specified trajectory tests break on every improvement and train you to ignore red.
• Budget for cost and latency as first-class metrics. A correct answer that costs a dollar and takes forty seconds is a failed answer in most products. Track tokens and time in the same eval that tracks correctness.
• Close the loop or the loop closes on you. Every production failure that doesn't become an eval case is a failure you've reserved the right to ship again.

Closing Thoughts

Agent evaluation feels exotic until you notice it's the oldest discipline in software engineering pointed at a new kind of target. You still write tests, still run them in CI, still gate releases on them, still monitor production and feed incidents back into the suite. What changes is that the system under test is probabilistic, so a single pass proves nothing (you measure distributions), the path is negotiable (you grade outcomes and watch trajectories), and some criteria aren't expressible in code (you bring in a judge, and then you have to evaluate the judge). Hold those three adjustments and the rest is the engineering you already know.

Which lands us, as these notes always seem to, on the same conclusion: a reliable agent is an architecture achievement, not a model one. A stronger base model raises your pass@1; it does not give you a regression suite, calibrate your judges, or turn last week's incident into next week's test. The teams shipping agents that people actually trust aren't the ones with the best model. They're the ones who treated evaluation as the test suite it is, and built the loop that makes the agent get better every time it gets something wrong.

References & further reading:
Anthropic: Demystifying Evals for AI Agents · τ-bench: A Benchmark for Tool-Agent-User Interaction · Sierra: τ-bench and the pass^k reliability metric · A Survey on Evaluation of LLM-based Agents · LLM Agent Evaluation Metrics: Tool Calling, Task Completion, Trajectory · An Empirical Study of LLM-as-a-Judge: How Design Choices Impact Reliability · Red Hat: Eval-Driven Development · OpenTelemetry for AI Observability

← Back to all posts