An AI agent saying “done” does not mean the work is done. It may have saved a file in the wrong location, called an unauthorized tool, or succeeded only by chance. Judging whether the final response sounds plausible can help with chatbots, but it is insufficient for an agent that changes external systems.
Anthropic describes an evaluation as an input plus grading logic and argues that agent evaluations must test the entire harness: model, prompt, tools, and environment. Google Cloud separates goal completion, response quality, tool use, hallucination, and safety. NIST notes that hidden tool calls and intermediate states require evidence and auditable traces.
This guide combines those ideas into five layers that can be applied to a small automation or a production workflow.
Layer 1: turn success into an observable contract
“Handle customer requests well” cannot be tested. Convert it into system states you can observe:
- The response contains the correct order and refund conditions.
- The ticket really changed to the intended status.
- No duplicate email was sent.
- Only approved tools and data were used.
- The task finished within time and cost limits.
Write each case as initial state → user goal → allowed actions → success state → forbidden conditions. Do not require one exact sentence when several responses could create the same correct state.
Include routine, edge, and adversarial cases: missing fields, conflicting requests, tool failures, insufficient permissions, long context, and hostile instructions. Start with a small representative set, then add a reproduction whenever production reveals a new failure.
Layer 2: check real outcomes deterministically
The final message is only a claim about the result. Whenever possible, inspect the result with code:
- Does the database contain the expected record and fields?
- Is the file in the correct location and schema?
- Was an API called exactly once?
- Do amounts, dates, and identifiers match source data?
- Did failure trigger a safe stop or rollback?
An LLM judge can assess an open-ended explanation, but existence, sums, permissions, and call counts should be deterministic. Such checks are fast, reproducible, and stable when the model changes.
Instead of grading “I scheduled the meeting,” inspect the calendar for participants, time zone, duplicates, and the conference link. Only then grade tone and clarity.
Layer 3: inspect the tool trajectory and forbidden actions
Two agents can reach the same output through very different risk profiles. Reading an approved CRM is not equivalent to searching an off-limits document store.
Capture the tools and order, summarized inputs and outputs, changed resources, retries, stop reason, and every human-approval checkpoint.
Avoid requiring one rigid sequence, which can reject a valid alternative. Define required checkpoints, acceptable path families, and tools that must never be called. Track success alongside zero forbidden actions, fewer unnecessary writes, and no approval bypass.
NIST also warns about grader gaming: an agent may exploit a loophole to maximize the score without achieving the intended goal. Keep answer patterns and test artifacts hidden, sample execution traces for human review, and never give the evaluation environment broader permissions than necessary.
Layer 4: separate quality, grounding, safety, and cost
A single aggregate score hides the reason for failure.
| Dimension | Core question | Preferred grading |
|---|---|---|
| Goal completion | Did the real task state become correct? | Deterministic checks first |
| Grounding | Are claims connected to evidence? | Source comparison + samples |
| Tool use | Were tools and arguments appropriate? | Trace rules |
| Safety | Were forbidden data and actions avoided? | Hard failure gate |
| Efficiency | Are latency, calls, tokens, and cost bounded? | Numeric thresholds |
| Response quality | Is the answer accurate, clear, and useful? | Rubric + human sample |
LLM judges are useful for scaling open-ended review, but they are not an absolute authority. Scores can shift with the judge model and prompt, and verbosity can be mistaken for quality. Use a specific rubric and examples, then calibrate judge scores against people regularly.
Critical safety failures should be gates, not averages. Unauthorized payment or personal-data exposure cannot be offset by a high writing score.
Layer 5: connect regression tests to deployment and production
Evaluation is not a launch-day event. A model, system prompt, tool schema, or retrieval index change can alter behavior.
A useful operating loop is:
- Run a fast critical set for every code change.
- Run the full routine, edge, and adversarial set for release candidates.
- Compare safety failures, cost, and latency—not only success rate.
- Route low-confidence and high-impact cases to human review.
- Turn sanitized production failures into new regression cases.
- Audit the evaluation set itself for leakage, duplication, and realism.
Run critical cases more than once to expose variance. A new model may raise the average while regressing on long context, a customer segment, or tool-error recovery, so retain segmented results.
A minimum viable evaluation spec
For each agent capability, begin with:
- five routine, three edge, and two adversarial scenarios;
- initial and expected final state for every case;
- deterministic checks and hard safety gates;
- allowed tools, forbidden tools, and approval requirements;
- three to five quality rubric items plus a human-review sample rate;
- latency, call-count, and cost ceilings;
- an owner and process for adding failure reproductions.
Write the release gate before seeing the result—for example, at least 95% goal completion, zero safety-gate failures, p95 latency under 20 seconds, and no regression on critical cases. Adjust the numbers to the risk, but do not move them after the score arrives.
Trust comes from evidence, not eloquence
A good evaluation does not ask whether the agent looks intelligent. It asks whether the intended state changed, approved methods were used, the system stopped when risk appeared, and the behavior repeats after the next update.
Separate final response, real state, action path, safety and efficiency, and regression. Start with small deterministic checks and feed incidents back into the suite. Trust in an agent comes from accumulated execution evidence, not an impressive demo.
Sources
- Anthropic — Demystifying evals for AI agents
- Google Cloud — Evaluate agents
- NIST — Security Considerations for AI Agents
- NIST — Cheating on AI Agent Evaluations
- NIST — Building Evaluation Probes into Agentic AI
Metrics and thresholds must reflect business risk, data sensitivity, and legal requirements. Keep independent security review and human approval for high-impact or irreversible actions.