How to evaluate and test AI agents
Build practical AI agent evaluations from representative tasks, explicit success criteria, deterministic checks and human review.
By Scrollport
In brief
Test an AI agent against the work it must complete, not against one ideal answer. Define observable success and unacceptable failure before testing, collect representative tasks, combine deterministic checks with human or model-based grading, and inspect tool choice, evidence, cost and safety as well as the final output. Keep the evaluation set alive as real failures and new use cases appear.
Test an AI agent against the work it must complete, not against one ideal answer. Define observable success and unacceptable failure before testing, collect representative tasks, and evaluate the route, evidence, cost and controls as well as the final output.
Evaluate the task, not one ideal answer
Agent tasks often allow several valid routes and outputs. A lead-research agent may find a different qualified company without being wrong. The evaluation should ask whether the result satisfies the agreed criteria, whether material claims have evidence and whether the agent stayed inside its authority.
Anthropic’s guide to agent evaluationsrecommends tasks grounded in realistic use and graders that measure the result and the path. Google’s agent evaluation guidesimilarly separates final response quality from trajectory evaluation.
Define success before testing
Write the acceptance contract before the run. Include the required result, source quality, freshness, format, spending limit, permission boundary and stopping condition. Also name the failures that must never pass, such as invented evidence, an unapproved write or a tool call above the human’s price threshold.
Use representative cases and edge cases
Start with real tasks from the intended workflow. Cover common requests, ambiguous inputs, missing data, provider failures, conflicting sources and a request that should be refused or sent to a human. Keep a separate holdout set so repeated tuning does not merely teach the agent the visible examples.
Combine deterministic, human and model graders
- Deterministic checks validate schemas, required fields, citations, budgets and forbidden actions.
- Human review judges usefulness, nuance and whether the result supports a real decision.
- Model graders can scale rubric-based comparison after calibration against human decisions.
OpenAI’s discussion of evaluation-driven developmentemphasises measuring the behaviour a product needs rather than relying on a generic model score. Keep the grader rubric narrow, test it on known good and bad examples, and review disagreements rather than treating any automated score as ground truth.
Test tool choice, cost and safety
For tool-using agents, inspect whether the agent selected a relevant capability, loaded the current contract, supplied valid inputs and used the result correctly. Measure unnecessary calls, retries, latency and total cost. Test the approval path, credential boundary and response to prompt injection or a malicious tool result.
Keep evaluations alive in production
Run the suite when instructions, models, tools or policies change. Add a production failure only after removing sensitive data and defining the expected behaviour. Track score, severity and cost by workflow rather than hiding a critical regression inside one average.
Observability supplies the next cases. Read what to trace and review, then keep evaluation results tied to the exact system version that produced them.