Measuring an AI agent in production

An AI agent does not fail like a server. It keeps answering, politely and with confidence, while the quality of its answers quietly erodes. That is what makes it treacherous: nothing flashes red. The only way to know whether it still does its job is to measure it continuously, on what actually matters. Here is what to watch, and how to put it in place.

Why does an agent that used to work start to drift?

Because nothing around it stays still. Your data moves: an updated catalog, a revised procedure, a brand-new customer case, and the agent answers wide of the mark without noticing. Usage moves too: people ask it questions it has never seen, often trickier than expected. And the ground shifts without warning: a model is updated or deprecated, an API returns a slightly different format. Each of these slips is tiny. Stacked together, they turn a reliable agent into a machine for plausible mistakes, and since it never crashes, no one sees it coming before a customer does.

Which metrics give false comfort?

The technical ones, the metrics that show up on a dashboard by themselves. An agent can be fast, available, never go down, and still be wrong one time in two. They measure the form, never the substance. Trusting them means mistaking a healthy agent for one that simply has not been caught out yet.

• Response time: useful for comfort, silent on correctness.
• Availability: the agent is running, which says nothing about what it is saying.
• Technical error rate: it counts crashes, not wrong answers delivered without flinching.

Which metrics truly matter?

The ones that speak to the agent's correctness and behaviour, not its plumbing. Four signals are enough to see the real problems coming.

• The correctness of the answers, evaluated continuously on a sample of real cases. This is the headline metric, the only one that tells you whether the agent does its job.
• The escalation rate to a human: too low and the agent settles cases it should hand off; too high and it is barely useful. Its drift is an early signal.
• The rate of unsupported answers, those cases where the agent asserts something your data does not back up. It is the direct measure of what makes it hallucinate.
• The share of tasks completed without human rework, and the cost per task. Together they tell you whether the agent is actually profitable, not just busy.

How do you set up continuous evaluation?

By stopping testing it once and for all on launch day. Evaluating an agent is not a final exam, it is a sensor you leave plugged in. The mechanism has five parts.

• A reference set of cases: a few dozen to a few hundred representative examples, each with its expected answer. It is the yardstick that stays put when everything else moves.
• A sample of real production: you continuously pull real interactions, not just lab cases.
• A three-level evaluation: automatic rules for what is checkable, a model that judges quality at scale, and a human review on a small sample to keep the whole thing honest.
• Thresholds and alerts: you decide in advance when quality has dropped too far, and the system warns you instead of waiting for a complaint.
• A correction loop: every mistake you catch feeds the reference set and the next fix. An evaluation that triggers nothing is worth nothing.

Why does observability change everything?

Because without it, you can see that an agent is wrong, never why. Observability means keeping a trace of every step: what the agent understood, the tools it called, the data it read, the decision it made. When an answer goes off the rails, you follow the thread instead of guessing. It is the difference between a one-hour fix and reopening the whole project. It is also what makes an agent auditable, a requirement that becomes standard the moment it touches decisions that matter. I make it a pillar of a system's reliability in production.

When should the agent hand off to a human?

As soon as it is unsure, and always on sensitive actions. A good agent knows its limits: faced with an ambiguous case, missing data, or an irreversible action, it escalates instead of forcing its way through. That is not a weakness, it is a safeguard, and its escalation rate is one of the numbers you watch. An agent that never escalates is not smarter, only more dangerous.

An agent you don't measure is not in production. It is a gamble.

If you have an agent in production and you only track its availability, you are not steering, you are hoping. Putting continuous evaluation, safeguards, and observability in place is exactly the work when I take a system from POC to production. A first conversation is often enough to know where to start, with no commitment: let's talk.