AI Agent Evaluation: How to Test Agent Skills
A practical guide to cases, controls, graders and release gates
AI agent evaluation checks whether an agent can complete one specific job reliably and safely before normal use. To evaluate agent skills, start with real tasks, test the decisions and side effects that matter, and use a release gate that matches the risk. A polished answer from one run is useful evidence, but it is not enough to trust an agent with customer messages, data changes, or money.
TL;DR: Begin with 8 to 12 cases from real work. Compare the intended agent setup with a no-skill baseline. Use code to check facts such as files, recipients, permissions, and tool arguments. Use a calibrated judge only for the parts that need human judgment. Block release when a critical boundary is crossed.
| Decision | What to test | Evidence to keep |
|---|---|---|
| Should the agent act? | Positive, negative, paraphrase and collision cases | Selection, abstention and false-activation rate |
| Did it complete the job? | Required facts, files, tool calls and final state | Assertions and output artifacts |
| Did it take a safe path? | Permissions, approvals, retries and side effects | Tool trace and boundary violations |
| Can a team rely on it? | Repeated runs, cost and latency | Pass rate, cost per success and p95 latency |
The useful question is simple: can this agent do this job under the conditions it will actually face? The Agent Skill Evals Manual contains a copyable version of the process below.
An AI agent evaluation should support a release decision
Every evaluation starts with a decision. Name the job, the person affected by a bad result, the evidence required for a release, and the boundary that ends the run. That prevents a team from collecting generic prompts that never settle an argument.
An agent that drafts a reply to a customer request needs more than fluent prose. It may need to identify the right customer, cite approved source material, create a draft in the correct thread, and stop before sending. Each requirement needs its own check.
This is where the YC Paper Club discussion on self-improving agent systems is useful. More tools and context can make an agent more capable. Evaluation tells a team whether those additions improve the work or add a new failure path.
Build an agent evaluation framework from real work
Start with the cases that have a visible cost when they go wrong. For each important decision, include an obvious positive, a paraphrase, a hard negative, and a collision where another workflow looks plausible. Add failure injection when the agent can call a tool or change state.
| Case | What it proves | Example |
|---|---|---|
| Positive | The correct workflow activates | A request that directly matches the documented job |
| Paraphrase | Normal wording variation does not break routing | The same request expressed with different vocabulary |
| Hard negative | The workflow stays out of unrelated work | A similar request that belongs to another owner |
| Collision | Competing instructions are resolved well | Two agent skills could apply, but one should lead |
| Failure injection | The agent recovers without crossing a boundary | A missing file, denied permission, timeout, or invalid payload |
Keep a release set separate from examples used to tune the instructions or judge. A case can still find regressions after it has informed implementation, but it no longer tells you much about new work.
Run the same important task with the intended setup, with no skill, and with a plausible but wrong skill when that control is available. Compare outcome, tool path, cost, latency, and critical violations. The SWE-Skills-Bench paper is a useful reminder that extra instructions can help some tasks and hurt others. Paired controls make the effect visible in your own environment.
Test output, trajectory and permissions separately
The final answer is only one part of the result. For tool-using agents, record the sequence of tool calls, arguments, approvals, retries, and state changes. A correct-looking response can still have a wrong recipient, an unsupported claim, an approval bypass, or an unnecessary destructive action behind it.
Use code for facts a machine can verify:
- A required file exists and validates against its schema.
- A recipient belongs to an allowlist and a reply targets the correct human message.
- A database change was staged, reviewed, or rejected as required.
- The expected tool received the intended arguments.
- A forbidden side effect did not happen.
Then use a semantic grader for the remaining questions, such as factual faithfulness, completeness, or whether the response answered the task. Give the grader a narrow rubric, a reference answer or source material where possible, and an abstain option. Sample its decisions against human labels and inspect the disagreements. Anthropic's guide to agent evals makes the same practical point: make the environment observable and choose the simplest evaluator that can assess the behavior.
Permissions deserve their own cases. Test denied access, ambiguous authority, prompt injection in retrieved material, retries after a timeout, and requests that mix safe work with an unsafe action. The ToxicSkills analysis offers examples of why third-party instructions and tool access need explicit review.
AI agent evaluation metrics that help a team decide
One score rarely describes an agent well. Match the measurement to the release decision.
| If the risk is | Track | A useful release question |
|---|---|---|
| Wrong routing | Precision, recall and false activation rate | Does the agent act only when it should? |
| Incorrect output | Paired pass rate and assertion failures | Does it improve the actual work over the baseline? |
| Unsafe action | Critical violations and approval bypasses | Did any run cross a boundary that blocks release? |
| Inconsistent behavior | Repeated-run pass rate | Does it work repeatedly under the same conditions? |
| Unusable economics | Cost per successful run and p95 latency | Can the team afford this reliability at normal volume? |
For a stochastic task, a single successful result is weak evidence. Repeat representative cases and report the conditions: model, tool versions, fixtures, grader version, and the number of runs. If every one of five runs must pass before a release, calculate and report that standard explicitly. It makes later comparisons possible when the model, tools, or instructions change.
A worked example: a draft-only customer-reply agent
Consider an agent that prepares a reply when a customer asks for help. The release decision is narrow: it may create a reviewable draft using approved material. It may not send a message, change customer records, or use an unverified source.
The initial suite could include a real support request, a paraphrased request, a similar sales inquiry that should route elsewhere, a request with an untrusted link, and a simulated API failure. Deterministic checks confirm the correct thread, source links, draft status, and absence of a send event. A human or calibrated semantic grader checks whether the draft answers the question faithfully.
That suite is small enough to run before every relevant change. When an incident occurs, investigate it first, then add the missing failure class to the evaluation set. The goal is a useful record of what the agent is allowed to do and how a team knows it did the work well.
Put the suite in the delivery loop
Run smoke tests before a small change lands. Run the paired suite before release. Review drift when you change models, tools, permissions, or source material. Production incidents are valuable input once the underlying behavior is understood.
Durable context helps here because the cases, reference artifacts, decisions, and corrections can stay connected. How teams keep AI-agent memory current explains the shared-memory side of that work. The Codex tutorial shows how instructions, skills, and verification can live in a repository. A Second Brain AI setup can provide the durable files and review loops behind the workflow.
Frequently asked questions
What is AI agent evaluation?
AI agent evaluation checks whether an agent can complete a defined task reliably and safely. It combines realistic cases, outcome checks, side-effect checks, repeated runs, and a release decision that matches the risk.
How many cases should an agent evaluation include?
Begin with 8 to 12 precise cases tied to decisions and failures that matter. Add cases when a release, incident, or user trace exposes a missing failure class. A smaller set with clear controls is more useful than a long list of generic prompts.
Should every AI agent evaluation use an LLM judge?
No. Use deterministic assertions for files, schemas, recipients, tool arguments, side effects, and approvals. Reserve model judges for semantic questions, then calibrate them against human labels.
What is the best first metric for an AI agent?
Use the metric that maps to the release decision. Routing may need precision and recall. A workflow with side effects may require zero critical violations. A variable task may need repeated-run pass rate, cost per success, and a latency budget.
When can a team release an agent without a full benchmark?
Release after a risk-appropriate suite shows the required behavior and protects critical boundaries. Low-risk internal helpers can begin with fewer cases. Agents that send messages, change data, handle money, or access secrets need stronger controls before routine use.
Start with one workflow
Pick a workflow with a visible failure cost. Write the release decision, create the initial case families, add a no-skill baseline, and make the important facts deterministic. Then use the Agent Skill Evals Manual to extend the suite with trajectory checks, judge calibration, reliability, cost, and security.