How I evaluate AI agents before they reach production

A Successful Demo Is The Beginning Of The Test

AI agent demos are unusually persuasive. A model reads an instruction, chooses the right tool, finds the right information, and produces a useful answer. The entire sequence may take less than a minute. It feels like the difficult part is finished.

What the demo actually proves is narrower: one version of the system completed one selected task under one set of conditions. It says little about what happens when the input is vague, a tool times out, permissions change, the conversation becomes long, or the same task is attempted for the hundredth time.

This distinction matters because an agent is not only a model response. It is a small distributed system. It has instructions, context, memory, tools, credentials, external dependencies, retry policies, and side effects. A failure in any of those layers can turn a reasonable model decision into a bad product outcome.

When I evaluate an agent, I am not trying to prove that it is intelligent. I am trying to measure whether the whole workflow is dependable enough for a specific job. That changes the questions from “Did the answer look good?” to “Did the system reach the correct outcome, through an acceptable path, without creating unacceptable risk?”

Start With The Job, Not The Model

An evaluation becomes useful only when the expected job is concrete. “Be a helpful assistant” is not testable. “Classify an incoming request, retrieve the correct account information, draft a response, and ask for approval before sending it” is much closer.

I describe the workflow as observable outcomes. What must be true at the end? Which facts must appear in the answer? Which system may be read? Which actions may be taken? Which action must never happen without approval? What should the user see when the task cannot be completed?

The acceptable behavior includes refusal and escalation. If an agent lacks a required identifier, guessing is not a partial success. Asking for the missing identifier may be the correct outcome. If a tool returns conflicting data, stopping and explaining the conflict may be better than selecting one result confidently.

This is why generic chatbot benchmarks rarely answer the production question. They can help compare model capabilities, but they do not know the permissions, business rules, recovery paths, or cost constraints of my workflow. The most valuable evaluation set comes from the actual job the agent is expected to perform.

Build A Dataset From Real Shapes Of Work

I begin with a compact set of representative tasks. It should include the common path, but common paths alone create a dangerously optimistic score.

The set needs normal variation: short requests, long requests, different wording, incomplete context, multiple items, and users who refer to earlier messages. Then I add boundary cases: missing records, duplicated names, stale data, unsupported file types, expired credentials, and requests that are almost within scope but not quite.

I also include adversarial cases that are realistic for the product. A retrieved document may contain text telling the agent to ignore its instructions. A user may request an action they are not authorized to perform. A tool result may contain malformed JSON or an unexpected field. An apparently harmless request may try to extract information from another account.

Production incidents and support tickets are particularly valuable additions. They represent failures that were important enough to reach a person. Each fixed incident should become a regression case, otherwise the same class of failure can quietly return after a prompt, model, or tool change.

A dataset does not need thousands of examples on the first day. A carefully chosen set of fifty tasks can reveal far more than hundreds of convenient examples. Coverage matters more than volume: different intents, risk levels, tool paths, failure modes, and conversation lengths.

Separate Outcome, Path, And Experience

A single pass or fail score hides too much. I evaluate an agent on at least three layers.

The first is the outcome. Did the user receive the correct result? Were required facts present? Was the requested change made exactly once? Did the system avoid making a change when it should have stopped?

The second is the path. Did the agent call an appropriate tool, use the correct arguments, respect permissions, and avoid unnecessary side effects? Two runs can produce the same final text while one used trusted account data and the other invented it. They are not equally successful.

The third is the experience. Was the answer understandable? Did it expose uncertainty? Did it ask a useful clarification instead of a vague one? Did the workflow finish within a reasonable time and provide progress when it could not finish immediately?

These layers help locate the fix. A correct tool result with a confusing answer is a presentation problem. A polished answer based on the wrong customer record is a correctness and security problem. A safe refusal for a supported task may indicate weak instructions or poor tool discovery.

Use Deterministic Checks Wherever Possible

Model-based grading is useful, but it should not replace checks that ordinary code can perform with certainty.

If an output must contain a valid account ID, I validate the ID. If the agent may only select one of six statuses, I check the enum. If it must cite a retrieved record, I verify that the cited identifier was present in the tool result. If sending requires approval, I assert that no send call occurred before the approval event.

Tool traces are excellent evaluation material. I can test the selected tool name, arguments, call order, number of retries, and whether a write action happened. For stateful workflows, I inspect the final database or external-system state instead of trusting the agent's description of what it did.

Deterministic checks are fast, cheap, and repeatable. They are especially important for permissions, money, identifiers, schemas, and irreversible actions. These are precisely the areas where “the answer sounds reasonable” is not a meaningful standard.

I reserve subjective grading for qualities that really are subjective: clarity, completeness, tone, and whether a clarification question is useful. Even there, a rubric with concrete criteria produces better results than asking a judge model whether an answer is simply good.

Judge Models Need Their Own Evaluation

Using one model to grade another is convenient. It scales better than asking a person to read every result, and it can detect semantic errors that string comparisons miss. It can also introduce a second layer of uncertainty.

A judge may prefer longer answers, reward confident language, or miss a subtle domain rule. It may grade differently after a model update. If it sees the candidate model's name, it may carry expectations that have nothing to do with the output.

I give the judge a narrow rubric and the evidence required to apply it. Instead of “rate this answer,” I ask separate questions: Are all claims supported by the supplied tool output? Did the response answer each part of the request? Did it disclose that one required record was unavailable?

Then I calibrate the judge against a small set scored by a person. Disagreements are informative. They show where the rubric is unclear or where automated grading cannot be trusted. High-risk cases still deserve human review, even if a judge model gives them a perfect score.

A model judge is a measurement instrument. Like any instrument, it needs calibration and known limits. Treating its score as objective truth merely moves the unexamined model behavior into the test suite.

Test Failures At The Tool Boundary

Many agent failures do not begin with bad reasoning. They begin with an ordinary integration problem.

I simulate timeouts, rate limits, authentication errors, empty responses, partial responses, and responses that succeed after a retry. I test a tool that performs an action but loses the acknowledgement. I return two records where the happy-path fixture returned one. I make a dependency slow enough to exceed the agent's normal execution window.

The correct reaction depends on the tool. A read-only search can often be retried. A payment, message, or public post cannot be repeated blindly because the first call may have succeeded. The agent needs a stable operation identifier, an idempotency mechanism, or a recovery path that checks the external state before acting again.

I also test whether tool errors are translated into useful product behavior. Raw stack traces and provider messages should not reach users. “Something went wrong” is not much better. The system should distinguish between a temporary problem, missing permission, invalid input, and a condition requiring human help.

An agent that succeeds only when every dependency is healthy is a prototype. A production workflow needs defined behavior for uncertainty.

Long Conversations Change The System

Single-turn tests miss one of the defining features of an agent: earlier messages affect later decisions.

I test conversations where the user corrects a fact, changes their mind, refers to “the second option,” or gives approval after several unrelated messages. I test whether old instructions are incorrectly carried into a new task and whether a summary preserves the details that matter.

Memory deserves separate cases. The agent should remember durable preferences when appropriate, but it should not turn every passing detail into a permanent fact. It should not expose information learned in a different account, channel, or permission context.

Context-window pressure is another boundary. As a conversation grows, some systems summarize or discard earlier content. I want to know whether constraints, pending approvals, identifiers, and unresolved questions survive that transition. A fluent final answer is not useful if it silently forgets that the user rejected the action ten messages earlier.

These tests often reveal architecture problems that prompt editing cannot solve. Important workflow state should live in a durable, structured place rather than depending entirely on the model reconstructing it from conversation text.

Measure Cost And Latency As Product Behavior

An agent can be correct and still be unsuitable for production.

I record total execution time, model tokens, number of model calls, tool calls, retries, and external API costs. A task that occasionally enters a reasoning loop can make the average look acceptable while producing a terrible tail latency and an unpredictable bill.

Percentiles are more useful than a single average. Users experience the slow runs, and infrastructure must absorb them. I look at the median, the slowest common cases, and the tasks that hit execution limits. I also separate time spent waiting for tools from time spent in model inference so the optimization targets the real bottleneck.

Quality and cost belong on the same chart. A larger model may improve difficult cases but add little value to routine classification. That can support routing: use a smaller model for constrained tasks and escalate ambiguous or high-risk cases. The decision should come from measured tradeoffs, not from assuming that one model should handle everything.

Budgets are also safety controls. Maximum steps, tool-call limits, and bounded retries prevent a confused agent from turning one request into an expensive loop.

Compare Changes Against A Stable Baseline

Agent development is full of changes that improve one example and damage another. A prompt becomes more cautious and starts refusing valid tasks. A new tool description improves selection but makes every simple request slower. A model upgrade follows instructions better but formats one integration's arguments differently.

I run the same evaluation set against the current production version and the candidate. The useful result is not only the candidate's total score. It is the diff: which cases improved, which regressed, which tool paths changed, and where cost or latency moved.

Evaluation inputs, expected outcomes, rubrics, prompts, tool schemas, and model settings should be versioned. Without that record, a score from last month cannot be compared honestly with a score from today.

I avoid releasing based only on a higher aggregate number. A one-percent improvement in common low-risk questions does not compensate for a regression that allows an unauthorized write. Results need to be segmented by capability and risk.

Production Monitoring Completes The Evaluation

No offline dataset contains the full creativity of real users or the full unreliability of real dependencies. Evaluation continues after release.

I monitor business outcomes and workflow stages: completion rate, escalation rate, corrections, abandoned runs, tool errors, repeated actions, and operations stuck in an intermediate state. User feedback is useful, but absence of a complaint is not proof of correctness. Some failures remain invisible to the person using the system.

Traces need enough detail to reconstruct a decision without leaking secrets into logs. A stable operation ID should connect the request, model calls, tool calls, approvals, and final state. Sensitive inputs can be redacted while preserving event type, timing, result category, and version information.

I sample successful runs as well as failures. Failure-only review misses silent problems: the workflow completed, but used three times as many steps; the answer was accepted, but cited stale data; the user manually corrected a field without reporting an issue.

The best production discoveries return to the offline suite. Monitoring finds a new failure shape, a person labels the expected behavior, and the case becomes a permanent regression test.

Release In Proportion To Risk

Not every agent needs the same gate. A private assistant that summarizes documents can tolerate different mistakes than an agent that changes customer data or publishes content.

I increase safeguards with consequence. Early versions may run in shadow mode, producing decisions without executing them. The next stage may require approval for every write. Later, low-risk actions can become automatic while unusual, expensive, or irreversible actions continue to stop for review.

A rollout can also be limited by users, task types, or traffic percentage. The important point is that rollback must be possible. Prompts, model versions, tool access, and routing rules should be deployable configuration with a known previous version.

Human-in-the-loop is not evidence that the agent failed. For many workflows it is the correct product boundary. Evaluation should measure whether the agent recognizes when it needs a person and provides that person with enough evidence to decide quickly.

Confidence Comes From Evidence, Not Fluency

A strong evaluation system does not promise that an agent will never fail. It makes the known behavior measurable and the unknown behavior easier to discover.

The practical foundation is straightforward: define the job, collect representative and difficult cases, inspect outcomes and tool paths, use deterministic assertions, calibrate subjective judges, inject dependency failures, test long conversations, and track cost and latency. Compare every change with a versioned baseline, then feed production observations back into the suite.

This work is less impressive on a screen than an agent completing a complex task in one attempt. It is also what turns that moment into a product people can depend on.

I trust an agent when I know more than how it succeeds. I want to know where it fails, whether the failure is visible, what it can safely retry, when it asks for help, and how quickly I can return to a known good version.

A demo shows that an agent can complete a task. Evaluation shows whether the system deserves to complete that task for a real user.