Your Agent Didn't Change, but It Got Worse: Release Gates, Drift, and Day-Two Evaluation

There's a failure mode that unsettles teams new to shipping agents: the deploy that changed nothing and made everything worse. You didn't touch the prompt. You didn't swap the model. You didn't retrain anything. And yet the resolution rate that sat at 82% for a month is now 71%, and the only thing that changed is the calendar. Traditional software doesn't do this. A function you don't edit computes the same thing forever. An agent sits on top of a model you don't control and a stream of user inputs you don't own, and both of those move. So measuring agent quality can't be a one-time gate you clear on the way to production. It's two distinct jobs: prove a change is good enough to ship, and then notice when a system you haven't changed is quietly decaying.

We've written before that evaluation is the test suite for a system that never runs the same way twice — what to measure (outcome, trajectory, tool use, cost), how to grade it (rules first, then a calibrated LLM-as-judge), and why consistency (pass^k) is a different axis than raw capability (pass@k). That's the foundation, and this piece assumes it. But that post is mostly about grading one run. This one is about the two things you do with that grading machinery over the life of a product: gating a release, and monitoring a fleet. They're the "day one" and "day two" of agent quality, and most teams build the first and skip the second right up until the second bites them.

Automated vs. Manual: A Division of Labor, Not a Choice

Before splitting the timeline, split the labor, because the "automated or manual?" question is where a lot of teams start and it's the wrong frame. You don't choose one. You use each for what it's uniquely good at, and the boundary between them is the single most important design decision in an eval program.

Automated evaluation — deterministic rules plus LLM-as-judge — is what makes evaluation a system rather than a heroic manual effort. It's cheap, fast, available on every commit and every production request, and it never gets tired at case four hundred. It's the only thing that scales to the volume a release gate and a live monitor demand. But it has a ceiling: rules can't judge nuance, and a judge is itself an unverified model until you measure it.

Human evaluation doesn't scale, and that's fine, because you're not asking it to. Its job is to be the ground truth the automated layer is measured against, and to see the things a rubric wasn't written to catch. A human reviewer reading fifty transcripts spots the new failure mode nobody anticipated, the answer that's technically correct but tonally wrong for your brand, the "corrupt success" that reached the right end state through a route you'd never approve. That's not overhead; it's how the automated graders earn their trust in the first place.

The relationship is a hierarchy, not a competition. Humans label a sample; that sample calibrates the judge; the calibrated judge grades everything; and humans periodically re-audit the judge to make sure it hasn't drifted from their standard. Automation gives you coverage; humans give you correctness; and correctness without coverage is a demo while coverage without correctness is a liability. Every mature stack runs all three, and the interesting question is never "which one" but "where does the boundary sit for this particular check" — a question you re-answer as the agent matures and the easy failures get squeezed out.

Before Release: Building a Gate That Actually Says No

The pre-release job has a clean success criterion: a change should not ship unless there's automated proof it's at least as good as what's live. That proof comes in two stages that you run in order — offline first, because it's cheap and catches the obvious regressions, then online-but-limited, because some failures only appear against real traffic.

Stage one: the offline regression gate

This is the eval suite from the previous post wired into CI as a release gate. Your domain eval set — built from real failures, starting at 20–50 cases and grown over time — runs on every change, each case executed multiple times so you're measuring pass^k consistency and not a lucky pass@1. The gate compares the candidate against the current baseline on outcome quality, and on the metrics teams love to forget: cost and latency. A prompt change that lifts resolution by two points while doubling token spend is not obviously a win, and only a gate that tracks all three lets you see the trade instead of discovering it on the invoice.

The highest-leverage offline technique is replaying production traffic. Take last week's real requests, run them through the candidate, and have a judge compare each new response against what the live system actually produced. This turns your most realistic possible test set — genuine user inputs — into a regression check for free, and it surfaces the changes that look great on your curated cases but quietly break the messy long tail of real usage.

Stage two: progressive delivery against live traffic

Offline evals pass and you still don't ship to everyone, because your eval set, however good, is a fixed snapshot and production is an infinite stream. So you expose the change gradually and let automated gates decide whether it earns more traffic. Three patterns stack into one pipeline, and the useful thing is knowing which question each one answers.

Shadow mode is the underused gem here. Because the candidate's output never reaches a user, you can point a risky change at 100% of real traffic with zero blast radius and let a judge tell you where it diverges from today's behavior before a single customer is affected. Canary then trades a small, bounded risk for the one thing shadow can't give you — real user reactions and real downstream effects — behind automated gates that roll back the instant a metric crosses a threshold. The through-line is that every stage is gated by the same automated evaluation, just pointed at a wider and riskier population of inputs each time. The offline suite gates the merge; shadow gates the canary; the canary gates the rollout. Nothing probabilistic reaches everyone without an automated proof at each step.

A rollback plan is part of the release, not a contingency. Version your prompts, your tool definitions, and your model pins the same way you version code, so "revert to the last-known-good agent" is one atomic action and not an archaeology project during an incident. An agent you can't cleanly roll back is an agent you can't safely roll out.

After Launch: The Two Ways a Static Agent Gets Worse

Here's the part that makes agents genuinely different from the services they otherwise resemble. You clear every gate, ship to 100%, and the metrics are green. Then, weeks later, with no deploy in the changelog, quality slides. There are two independent culprits, and telling them apart is the whole game because they have different fixes.

• The model changed under you. If you call a hosted model behind an API, the provider can update the weights beneath a stable version string, or you pinned to a snapshot that's being deprecated and silently rerouted. Same prompt, same code, different behavior. This is real model drift, and the only defense is a canary eval set you re-run on a schedule against the live endpoint so you notice the day the model's answers move.
• The world changed under you. Your code is frozen but your inputs aren't. Users bring new topics, new slang, a new product line, a seasonal surge, a category of question your agent has never seen. The agent didn't get worse at the job you evaluated; the job quietly became a different job. This is distribution drift, and it's the failure that offline evals structurally cannot catch, because your eval set was frozen the day you wrote it.

It's worth being precise about drift, because "drift" gets used for two distinct things that call for different responses:

Data drift means your evaluation is aimed at the wrong target; concept drift means your target moved. Both present as "quality dropped," and both are invisible to a gate that only runs before release. The only way to see them is to keep evaluating in production, continuously, forever.

Monitoring in Production: Watch the Inputs, Sample the Outputs

Online evaluation is where the microservice framing pays off literally. An agent emits OpenTelemetry traces like any distributed system — a parent-child span tree with a span for every LLM call, tool execution, and sub-agent handoff — and that trace data is the substrate everything else runs on. On top of it you build monitoring in two layers, because you can't afford to grade every run with a judge but you can't afford to grade none of them either.

• Cheap signals on 100% of traffic. Fast, deterministic heuristics on every request: latency and cost per run, error and tool-failure rates, guardrail trips, output-length anomalies, refusal and fallback frequency. These are nearly free and they catch the loud failures immediately.
• Expensive signals on a sample. Run an LLM-as-judge on a small slice of live runs — commonly 5–10% — grading for faithfulness, policy compliance, and resolution, the same open-ended dimensions the offline judge scores. Sampling keeps the cost bounded while still giving you a statistically meaningful read on quality trends.

Add to that the input-side signal that offline evals can't produce: drift detection on the requests themselves. Because your agent already embeds text for retrieval and semantic caching, you have embeddings for free, and you can watch the distribution of incoming queries move over time. The methods range from cheap-and-crude to sophisticated:

• Statistical distances — Population Stability Index (PSI), KL divergence, or a KS test on tractable features. Fine for simple scalar or token-frequency signals; largely blind to high-dimensional semantic change on their own.
• Centroid movement — track the mean embedding of recent queries and measure how far it drifts (Euclidean or cosine) from a reference window. Simple and interpretable, with the caveat that thresholds need tuning per embedding model.
• A domain classifier — train a lightweight binary classifier to tell reference traffic from current traffic and read its ROC AUC. If it can easily distinguish "last month" from "this week," your inputs have moved. Evidently's own comparison lands on this as the sensible default: it tunes intuitively and stays consistent across embedding models.

The mistake is to alert on any one of these in isolation. Input drift alone might just mean traffic got easier; a judge-score dip alone might be sampling noise. The signal worth paging on is the joint condition: input distribution shifted and your sampled eval score dropped. That conjunction is the fingerprint of drift actually hurting quality, and it's far more specific than either half. Wire your alert to the and, not the or, and you get a monitor that fires when it matters and stays quiet when it doesn't.

Multi-agent systems add a slower, sneakier variant. In long-running agents, small behavioral deviations can compound across many steps and sessions — a gradual "agent drift" that no single trace looks wrong enough to flag. That's an argument for watching trends on aggregate metrics over days and weeks, not just thresholds on individual runs. The run that finally fails is rarely the run where the rot started.

Closing the Loop, Which Is the Whole Point

Monitoring that only produces dashboards is monitoring you'll eventually stop reading. The payoff comes from a feedback loop that's identical in spirit to the incident-to-regression discipline every good engineering team already runs. Production surfaces a failure — a drifted input the agent fumbled, a judge-flagged bad answer, a human-reviewed transcript that made you wince. You capture that real example, add it to the eval set, and it becomes a permanent regression case. The next candidate that would reintroduce that failure gets caught at the offline gate, before it ships. That's the flywheel: the offline suite protects you from repeating known failures, and production is the endless source of new ones. An eval set maintained this way is a living record of every mistake you've decided never to make again.

This is also why the two jobs aren't really separate — they feed each other. The pre-release gate is only as good as the cases in it, and the best cases come from post-launch monitoring. Skip the monitoring and your gate slowly goes stale, checking for yesterday's problems while today's drift walks right past it. The teams whose agents stay good in production aren't running a better gate or a better monitor; they're running the loop that keeps each one fed by the other.

A Field Checklist

• Gate the merge on an offline regression suite, cost and latency included. "At least as good as live, and not more expensive" is the bar. Run each case multiple times so you're gating on consistency, not a lucky sample.
• Replay real production traffic through every candidate. Your most realistic test set already exists in your logs; a judge comparing new-vs-current responses on last week's requests catches what curated cases miss.
• Never jump straight to 100%. Shadow for zero-risk sanity, canary for bounded real-world risk, A/B when you need to know which is genuinely better — each gated by the same automated evals.
• Assume the model will change under you. Re-run a canary eval set against the live endpoint on a schedule, so provider-side model drift shows up as a failing check and not a customer complaint.
• Monitor inputs, not just outputs. You already have embeddings; watch the query distribution for drift and pair it with a sampled judge on outputs.
• Alert on the conjunction. Page on "inputs drifted AND eval score dropped," not on either alone, and you'll trust your alerts enough to act on them.
• Version everything for instant rollback. Prompts, tool definitions, and model pins under version control, so reverting to the last-known-good agent is one action.
• Feed every production failure back into the offline set. A failure that doesn't become a regression case is a failure you've reserved the right to ship again.

Closing Thoughts

The uncomfortable truth about agent quality is that "we evaluated it and it passed" is a statement about a moment, and moments expire. You built the agent against a model and a user base that will both have moved by next quarter, and neither of them will send you a deploy notification when they do. So measurement can't be an event you clear on the way to launch; it has to be a standing capability that runs before every change and behind every request, with a human in the loop often enough to keep the automated graders honest.

None of this is exotic. Progressive delivery, regression gates, production monitoring, incident-to-test feedback loops — this is release engineering, the discipline that made ordinary software trustworthy, pointed at a system whose two foundations happen to shift on their own. The agent teams that sleep at night aren't the ones with the model that scored highest on launch day. They're the ones who assumed that score would decay, and built the gate and the monitor that catch it when it does.

References & further reading:
Anthropic: Demystifying Evals for AI Agents · Evidently AI: Five methods to detect drift in ML embeddings · AWS Prescriptive Guidance: Detecting drift in production applications · Arize: LLM Observability and Drift Detection · τ-bench: pass^k and agent reliability · Arthur: Best Practices for LLM Testing Before Deployment

← Back to all posts