How to Evaluate Agent Skills
A practical guide to cases, controls, graders and release gates
Agent skills should be evaluated as controlled changes to an agent, not as prompts that look good once. Use realistic tasks, run them with and without the skill, assert the facts a machine can check, and inspect the failures that remain. A useful suite tells you when a skill triggers, whether it helps, and whether it creates an unsafe or expensive path to the result.
TL;DR: Start with 8 to 12 cases from real failures. Include positives, paraphrases, hard negatives and collisions. Run a no-skill baseline, use deterministic checks for required facts, repeat variable cases, and block release on critical safety failures.
| If you need to know | Test it with | Evidence to keep |
|---|---|---|
| Does the right skill activate? | Triggering, negative and collision cases | Selected skill, abstention and false activation rate |
| Does it improve the work? | Same task with the skill, without it and with a wrong-skill control | Paired pass-rate delta, cost and latency |
| Did it do the required work? | File, schema, tool argument and final-state assertions | Assertion output and artifacts |
| Is the result reliably good? | Repeated runs plus a calibrated semantic grader | pass^k, judge agreement and disagreements |
| Did it cross a boundary? | Failure injection and adversarial cases | Critical violations, approvals and side effects |
Define the decision before you write cases
An eval exists to support a release decision. Name the skill owner, the user-facing job, the expensive failure it should prevent, and the boundary that must never be crossed. This stops the team from collecting generic prompts that cannot settle an argument later.
For example, an email-sending skill should not be judged only on whether the prose sounds natural. It may need to select the correct recipient, use an approved reply target, avoid reply-all, leave a traceable draft, and stop before sending when approval is absent. Each of those is a different assertion.
The Agent Skill Evals Manual includes a copyable specification for recording that contract before a run.
Build a dataset from real decisions and failures
Start with a small set of cases that are expensive when wrong. A good initial mix has one obvious positive, one paraphrase, one hard negative and one ambiguous or competing-skill case for each important decision. Add a failure-injection case wherever a tool can change state.
| Case family | What it proves | Example |
|---|---|---|
| Positive | The expected skill is selected | A request that directly matches the skill’s documented job |
| Paraphrase | Triggering survives normal wording variation | The same request expressed with different vocabulary |
| Hard negative | The skill stays out of unrelated work | A similar-looking task that belongs to another workflow |
| Collision | Routing resolves competing instructions | Two skills could plausibly apply, but only one should lead |
| Failure injection | Recovery preserves the boundary | An API timeout, missing file, invalid payload or denied permission |
Keep a release set separate from examples used to tune a prompt or a judge. If a case has influenced the implementation, it can still diagnose regressions, but it no longer independently estimates how the skill performs on new work.
Run a no-skill baseline
A skill earns maintenance cost only when it creates a measurable improvement. Run the same task in three conditions when practical:
- With the intended skill.
- With no skill.
- With a wrong or competing skill.
Compare the final outcome, tool path, cost, latency and critical violations. This turns “the skill seems useful” into a specific claim. It also catches a common failure: an instruction file adds tokens and constraints but does not change the result, or makes it worse.
The 2026 SWE-Skills-Bench paper is a useful warning. Its authors report that skill effects vary and can be negative when instructions do not match the task. Treat that as a reason to keep paired controls, not as a universal score to copy.
Grade deterministic facts before semantic quality
Most high-value requirements do not need a language-model judge. Check them with code first: required files exist, JSON validates, a recipient matches an allowlist, a tool was called with the right arguments, an approval was recorded, or an unwanted state change did not occur.
Use a semantic judge only for the remaining questions, such as whether a response is faithful to provided evidence or whether an explanation is complete. Give the judge a narrow rubric, references where possible, a clear scoring scale, and an abstain option. Then compare a sample of judge labels with human labels and investigate disagreements.
Anthropic’s 2026 guide to agent evals makes the same practical distinction: make the environment observable and use the simplest reliable evaluator for the behavior you care about.
Check trajectories and side effects separately
Final text can look correct while the agent reaches it through a risky path. For tool-using skills, capture the sequence of tool calls, arguments, approvals, retries and side effects. Then decide what is required, forbidden and merely one valid alternative.
For a resource-page skill, the final page must be present, but the route also needs a registry entry, capture configuration, sitemap, IndexNow entry and prerender link. For a finance or outreach skill, the final draft may be good while a wrong recipient, duplicate send or missing approval is a release blocker. The evaluator should observe the parts that matter to the user, not force one arbitrary internal path.
If you are building a repeatable technical workflow, our Claude Code tutorial shows how durable instructions, skills and verification fit together in a repository.
Repeat stochastic cases and report reliability honestly
One clean run is a demo. Run important stochastic cases several times and report pass^k, the chance that all k runs pass under the tested conditions. Include model, skill, tool, fixture and grader versions so a later result can be compared rather than guessed at.
Track cost per successful run and p95 latency alongside success. A skill that improves accuracy but doubles the cost or introduces long retries may still be the wrong default for routine work. Set thresholds according to risk: a spelling helper and a payment-changing workflow should not share the same gate.
Add security cases before the skill reaches routine use
Test for prompt injection, unsafe tool use, secret exposure, approval bypass, memory poisoning, duplicate retries and destructive actions hidden inside otherwise plausible work. A critical violation should veto a release even if every style or completion metric is strong.
The ToxicSkills analysis from Snyk is relevant here as a practitioner signal: skills expand an agent’s behavioral surface, so provenance and runtime checks matter. Community and vendor reports can help find attacks to test. Use direct system evidence and controlled tests to decide whether your own implementation is safe.
Put the suite in the delivery loop
Run smoke tests before a change lands, the full paired suite before release, calibration and drift review weekly, and a broader security or migration review when you change models, tools or permissions. Every production incident should become a candidate eval case after the underlying behavior is understood.
This is where a local AI system has an advantage. The Second Brain AI setup is designed around durable context and review loops, which makes it easier to keep the cases, artifacts and decisions connected to the work that produced them.
Frequently asked questions
How many cases should an agent skill eval have?
Begin with 8 to 12 precise cases tied to decisions and failures that matter. Expand the suite when a new release, incident or user trace reveals a missing failure class. A smaller suite with clear controls is more useful than hundreds of generic prompts.
Should every agent-skill eval use an LLM judge?
No. Use deterministic assertions for files, schemas, recipients, tool arguments, side effects and approvals. Reserve model judges for semantic questions, then calibrate them against human labels.
What is the best first metric?
Use the metric that maps to the release decision. Triggering may need precision and recall. A side-effecting workflow may need zero critical violations. A stochastic task may need pass^k, cost per success and a latency budget. There is no single score that covers all of these.
When can I publish a skill without a full benchmark?
After a risk-appropriate suite shows the required behavior and protects critical boundaries. Low-risk internal helpers can start with fewer cases. Skills that send messages, change data, handle money or access secrets need stronger controls before routine use.
Start with one skill this week
Pick one skill with a visible failure cost. Write the release decision, create four case types, add a no-skill baseline and make the important facts deterministic. Then use the Agent Skill Evals Manual to add the remaining layers: trajectory checks, judge calibration, reliability, cost and security.