A production AI agent needs two connected systems: offline evaluation to decide whether a version is safe and useful enough to release, and runtime monitoring to detect failures that test data did not predict. Measure the whole workflow, not only the final answer. That means checking task completion, tool selection, arguments, approvals, side effects, policy compliance, latency, cost, and recovery behavior.
The objective is not a universal accuracy score. It is evidence that the agent meets a defined operational contract for a defined population and risk tier. NIST describes trustworthy AI as dependent on reliable test, evaluation, verification, and validation. In practice, the evidence comes from representative cases, explicit rubrics, calibrated graders, trace inspection, controlled releases, and production feedback that becomes new regression data.
Key takeaways
Define success and unacceptable failure before choosing models or dashboards.
Evaluate final outcomes and the trajectory: retrieval, reasoning-visible events, tool calls, handoffs, approvals, and side effects.
Use deterministic checks where possible, human judgment where necessary, and model graders only after calibrating them against human labels.
Segment results by task type, risk, language, customer group, tool, and version; an average can hide a severe local failure.
Turn incidents, overrides, escalations, and near misses into durable test cases and release gates.
Start with an evaluation contract
Write a one-page contract for the workflow. State the user’s goal, allowed actions, required evidence, completion conditions, escalation conditions, prohibited behavior, and maximum operational budgets such as turns, time, or spend. Separate quality requirements from safety constraints. A support agent might be required to identify the right account, use only approved policy, classify the issue, and either resolve it or escalate with a complete handoff. It must never expose another customer’s data or perform a restricted account change.
Build the offline evaluation stack
1. Assemble representative and adversarial cases
Start with real, permissioned examples that reflect normal volume, ambiguity, incomplete data, and common exceptions. Remove or protect sensitive information. Add rare but costly cases intentionally: conflicting records, unavailable tools, stale knowledge, prompt injection inside an attachment, malformed tool output, duplicate events, timeouts, and a request that must be refused or escalated. Tag each case by source, scenario, risk, expected path, and affected policy so results can be sliced rather than averaged blindly. Keep a held-out set for release decisions and prevent examples from leaking into prompts or tuning data.
2. Use a hierarchy of graders
Prefer the simplest valid grader. Schema validation, exact field checks, database-state comparison, policy rules, and executable test oracles are more reproducible than subjective scoring. Use subject-matter experts for judgments that depend on nuance. Model graders can scale review of relevance, completeness, or policy adherence, but first compare them with labeled human examples, document disagreement patterns, and recheck them when the model or rubric changes. A grader should return a decision and structured reason, not an unexplained score.
3. Grade trajectories, not just messages
Two runs can produce the same answer with very different risk. One may retrieve an authorized record and use the correct read tool; another may query too broadly, expose sensitive fields, retry excessively, and then happen to answer correctly. Trace graders should test whether the agent chose an appropriate tool, supplied valid arguments, honored handoff and approval rules, stayed within budgets, and verified the resulting state. Do not require one exact path when several are legitimate, but do prohibit dangerous shortcuts. Current OpenAI guidance likewise positions traces as an end-to-end record of model calls, tools, guardrails, and handoffs.
4. Create release gates
Compare every material change against the current production baseline: model, prompt, tool description, schema, retrieval index, policy, routing, or memory behavior. Define hard gates for safety-critical failures and scenario-level thresholds for quality and operations. Review a sample of passes and failures manually. A launch decision should include known limitations and confidence by segment, not only a headline pass rate. Use shadow traffic or a constrained canary where appropriate, and preserve a fast rollback path.
Instrument the production workflow
Create one trace per end-to-end run and spans or events for model calls, retrieval, tool execution, handoffs, approvals, queue waits, and external API calls. Capture a run ID, conversation or task ID, tenant, agent version, prompt and policy version, model, tool version, latency, token usage, outcome, and error class. For side effects, record the target’s stable identifier and confirmation status. Avoid indiscriminately logging prompts, tool arguments, or results: OpenTelemetry’s GenAI attributes warn that these fields may contain sensitive information. Apply redaction, sampling, access control, and retention before telemetry reaches a vendor. Pin the semantic-convention version because GenAI conventions continue to evolve.
Monitor four layers of signals
Outcome quality: task success, correct escalation, verified side effect, rework, human override, user correction, and unresolved outcome.
Trajectory quality: retrieval relevance, wrong-tool selection, invalid arguments, policy violations, repeated loops, unnecessary handoffs, and skipped approvals.
Operational health: end-to-end and component latency, timeout and retry rates, tool/API errors, queue depth, maximum-turn exits, token use, and cost per completed task.
Safety and control: unauthorized access attempts, prompt-injection indicators, sensitive-data detections, approval rejections, unusual destinations, transaction-limit blocks, and kill-switch activations.
Close the loop after incidents and feedback
When a material failure occurs, first contain side effects: pause the workflow, revoke or narrow credentials, stop queued work, and preserve evidence. Identify all records touched by the same version or failure condition and reconcile them against systems of record. Then reduce the failure to the smallest reproducible case, add it to the evaluation set, fix the appropriate layer, and run the full regression suite. The fix may belong in authorization, a tool schema, data quality, workflow code, a prompt, or human procedure; changing the model is not always the answer.
Implementation checklist
Write the workflow contract, risk tier, allowed actions, completion criteria, and escalation conditions.
Create a versioned dataset with normal, edge, adversarial, and refusal cases plus segment tags.
Implement deterministic graders first; document rubrics and calibrate human or model graders.
Trace model, retrieval, tool, handoff, approval, error, and side-effect events with version identifiers.
Set scenario-level release gates, manually inspect samples, and compare changes with the production baseline.
Deploy through shadow, canary, or constrained permission stages and keep rollback and kill controls tested.
Build alerts with owners and playbooks; review quality, safety, operations, and business outcomes on a fixed cadence.
Feed incidents, overrides, escalations, and changed upstream behavior back into tests.
Limitations and when monitoring is not enough
An eval set is a sample, not proof of universal reliability. Production labels can be delayed or biased, users may not report quiet errors, and model graders can share blind spots with the system they judge. Telemetry can also expose sensitive content if collected carelessly. If an outcome cannot be verified, a harmful action cannot be reversed, and qualified humans cannot review ambiguous cases, the workflow may not be suitable for autonomous operation. Reduce the action scope or use the model only to draft and organize evidence.
Frequently asked questions
How many evaluation examples does an AI agent need?
There is no universal number. Coverage matters more than an arbitrary count. Begin with examples for every critical path and failure mode, then add cases until new reviews stop revealing major unrepresented categories. Track confidence by scenario and risk, and expand the set continuously from real failures and changes in task mix.
Can an LLM grade another agent reliably?
It can be useful for well-defined subjective criteria, but it should be calibrated against representative human labels. Test agreement, false passes, false failures, ordering bias, and sensitivity to rubric wording. Use deterministic checks for facts and system state whenever possible, and keep expert audits for high-impact decisions.
What should trigger an immediate rollback?
Predefine conditions based on the workflow’s risk: unauthorized access, confirmed sensitive-data disclosure, execution without required approval, wrong-target or duplicate transactions, material policy violations, or a sharp loss of verification coverage. A rollback rule needs an owner, tested mechanism, and reconciliation plan, not just a dashboard threshold.
Should production prompts and tool results be logged?
Only when there is a defined diagnostic or audit need and the data handling is acceptable. Prefer structured metadata and identifiers over full content. Redact secrets and sensitive fields, restrict access, set retention, and sample where possible. Confirm that logging vendors and regions meet the same obligations as the application.
Design the evidence before expanding autonomy
Book a System Audit to turn one workflow into an evaluation contract, trace design, release gate, and production response plan. The 24/7 customer support automation use case is a useful reference for mapping resolution, escalation, knowledge, and handoff signals without treating conversation volume as proof of quality.
Sources
Reviewed against current NIST TEVV material, the NIST Generative AI Profile, current OpenAI agent-evaluation and trace-grading guidance, and OpenTelemetry conventions.
OpenOperative Editorial Team
Technical Editorial Team