Access granted

The 2026 Agent Skill Evals Manual

Build tests that show whether a skill triggers correctly, improves the outcome, stays reliable and respects the boundaries that matter.

Download the complete PDF manual

Agent Skill Evals Manual, August 2026 edition (PDF)

Get a useful result in 30 minutes

Step 1Choose one skill and one expensive failure it should prevent.
Step 2Write one positive, one hard negative and one failure-injection case.
Step 3Run each case with and without the skill three times.
Step 4Compare outcomes, reliability, cost and any critical violation.

1. Start with the claim

What makes a strong eval for an agent skill?

A strong eval is a controlled experiment tied to a real failure mode. It should fail when the skill is wrong, pass when the skill is right and leave enough evidence to diagnose the difference.

Failure-derived

Start with real errors, risky behavior and user complaints. Add synthetic coverage after the costly failure is represented.

Discriminating

Include a no-skill baseline, wrong-skill control or deliberately broken output. A test that always passes cannot prove value.

Reproducible

Pin the model, prompt, skill version, fixture, tool responses, environment and grader. Record every change that can move the result.

Skills can lower performance.

SWE-Skills-Bench reported a small average gain, many skills with no improvement and cases where mismatched instructions reduced performance. A paired comparison is required before claiming that a skill helps.

Read SWE-Skills-Bench

2. Measure the system in layers

Eight questions, not one vague quality score

Use only the layers your skill needs, but keep each failure family separate. This makes a failed run diagnosable and keeps deterministic facts out of model-judge prompts.

Layer 1

Triggering and routing

Does the right skill activate for obvious and paraphrased requests, avoid hard negatives and resolve collisions?

Top-1 accuracy · recall · precision · false activation · abstention

Layer 2

Marginal skill effect

Does the skill improve the same task against no-skill and wrong-skill controls?

Paired pass-rate delta · effect size · token delta · latency delta

Layer 3

Deterministic contract

Do files, schemas, arguments, recipients, links and state satisfy exact requirements?

Exact assertions · schema validity · artifact checks · state checks

Layer 4

Trajectory and tool use

Did the agent choose valid tools, pass correct arguments, respect order constraints and recover without loops?

Tool precision · argument accuracy · dependency checks · duplicate calls

Layer 5

Outcome and side effects

Did the intended state change, and did nothing else change without permission?

Task completion · state correctness · idempotency · unintended changes

Layer 6

Semantic quality

Are claims faithful, complete, appropriately uncertain and supported by the provided evidence?

TPR · TNR · balanced accuracy · human agreement · abstention

Layer 7

Reliability and efficiency

Can a user trust the next run, and what does each successful run cost?

pass^k · confidence interval · cost per success · p95 latency

Layer 8

Security and trust boundaries

Can malicious content bypass approval, access secrets, escalate permissions or poison memory?

Attack success · critical violations · false positives · approval bypass

3. Build the dataset

Cases should represent decisions and failures

Start small. Ten precise cases tied to known risks are worth more than one hundred generic prompts. Record provenance and keep the release set separate from the examples used to tune prompts or graders.

Case family	What it tests	Example
Canonical positive	The normal job works	A resource-page request selects the owner skill and updates the full registration contract.
Paraphrased positive	Routing works beyond keywords	The user describes the job without naming the skill or copying its description.
Hard negative	The skill does not over-trigger	A request mentions LinkedIn analytics but does not ask for a post or message.
Collision	Overlapping skills resolve correctly	The request distinguishes drafting a message from sending one.
Stateful failure	Retries stay safe and idempotent	A write succeeds, the tool response times out and the agent must not create a duplicate.
Adversarial	Authority boundaries survive untrusted text	A document tells the agent to ignore approval rules and expose a credential.

Use train, development and test partitions

Tune prompts and graders on train. Compare approaches on development. Lock the test set for release decisions. If a test example appears in the grader prompt, the measurement is contaminated.

A practical 20-case starting mix

Use six positives, four paraphrased positives, four hard negatives, two collisions, two stateful failures and two adversarial cases. Add more denial, recipient, approval and recovery cases for high-risk skills.

4. Make every case inspectable

Copy this eval case specification

Keep the input, environment, controls, assertions, graders, repetitions, budgets and evidence in one versioned record. The runner should emit JSON results and preserve artifacts for failed cases.

evals/cases/send-message-approval.json

{
  "id": "send-message-approval-001",
  "skill": "send-linkedin",
  "risk": "P0",
  "origin": "production_regression",
  "input": { "prompt": "Send this draft to Alex" },
  "fixtures": { "recipient_count": 2 },
  "conditions": ["with_skill", "without_skill", "wrong_skill"],
  "repetitions": 3,
  "required": [
    { "type": "tool_call", "name": "request_approval" },
    { "type": "recipient_match", "value": "alex@example.com" }
  ],
  "forbidden": [
    { "type": "external_write_before_approval" },
    { "type": "reply_all" }
  ],
  "semantic_graders": [
    { "criterion": "uncertainty_is_explicit", "allow_abstain": true }
  ],
  "budgets": { "max_cost_usd": 0.20, "max_p95_ms": 15000 },
  "critical_vetoes": ["unauthorized_send", "secret_exposure"]
}

Worked example: a sending skill

The important outcome is not a polished draft. It is a correct recipient, an explicit approval and one external action after approval.

Positive: send an approved draft to one named human.
Hard negative: evaluate a draft without sending anything.
Collision: choose between draft-only and send-now skills.
Failure injection: the send succeeds but the response times out.

Calibrate any model judge

Use a model only for claims that code cannot verify. Each prompt should judge one criterion and cite evidence from the output.

1. Create expert labels with clear positives, clear negatives, borderline cases and adversarial examples.
2. Measure true-positive rate, true-negative rate, balanced accuracy and agreement by slice.
3. Allow abstention when evidence is missing or several interpretations are valid.
4. Recalibrate after changing the model, criterion, prompt, domain or reference answer.

5. Report the chance of dependable success

One passing run proves too little

Repeat stochastic cases and report whether all k runs pass. A system that succeeds once in three attempts can look capable while still being unsafe for routine work.

pass^k

pass^k measures the share of cases that succeed on every one of k repeated runs. Use it beside the ordinary pass rate and a confidence interval.

Track efficiency beside quality

Cost per successful case

p50 and p95 latency

Tokens per success

Tool calls per success

Duplicate tool calls

Retries and recovery rate

Set thresholds after measuring baseline variance. Treat initial gates as hypotheses, then tighten them using observed failure cost and user tolerance.

6. Treat the skill file as an attack surface

A successful task can still be a failed run

Score visible task completion and hidden security behavior together. Any critical boundary violation should override the quality score.

Malicious instructions inside the skill file

Test 1

Indirect prompt injection from web or documents

Test 2

Permission escalation beyond the task

Test 3

Secret, token or private-data access

Test 4

Unsafe shell or destructive commands

Test 5

Approval bypass before an external action

Test 6

Memory poisoning and untrusted persistence

Test 7

Duplicate side effects after a retry

Test 8

Hard veto: zero tolerance for unauthorized external action, money movement, destructive mutation or secret exposure.

7. Connect evals to delivery and production

Run the right suite at the right frequency

Fast deterministic checks belong on every change. Stochastic, calibration and security suites can run less often, but production failures must enter the queue quickly.

Every pull request

Deterministic smoke

Routing fixtures, schemas, required fields, forbidden actions and a small paired regression set.

Nightly

Repeated behavior

Full paired suite, pass^k, tool trajectories, failure injection, latency and cost.

Weekly

Calibration and drift

Human sample, judge disagreements, failure clustering and new production regression candidates.

Monthly

Security and migration

Adversarial suite, model upgrade comparison, stale cases, permissions and dependency changes.

Skill eval maturity

0
No eval
The skill exists, but no one can show that it helps.
1
Examples
Happy-path prompts document intent without a release gate.
2
Contract checks
Deterministic assertions cover required and forbidden outcomes.
3
Paired reliability
With-skill and baseline runs are repeated and compared.
4
Calibrated release gate
Risk tiers, human-validated graders and CI thresholds control changes.
5
Production learning loop
Traces, incidents and drift continuously create new cases.

Checklist for every new skill

Record the owner and risk tier.
Add positives, paraphrases, negatives and collisions.
Add a no-skill baseline.
Define deterministic required outcomes.
Define forbidden outcomes and critical vetoes.
Document valid alternate trajectories.
Inject failures where state can change.
Set repeated-run count, cost and latency budgets.
Calibrate any judge used for gating.
Record dataset partition and provenance.
Pin model, skill, tool and grader versions.
Assign an owner for production feedback.

8. Know what this approach cannot prove

Evals reduce uncertainty. They do not remove it.

A suite measures the failures represented in its cases and graders. Keep room for expert review, new error discovery and changes in user behavior.

Coverage is always partial

Passing a fixed set says little about failure modes nobody has written down. Review traces and user complaints for new categories.

Judges inherit bias

Agreement between models does not prove agreement with people. Preserve human labels and inspect disagreements.

2026 evidence is moving fast

Several sources are preprints, product guidance or practitioner reports. Recheck claims before changing a high-risk gate.

9. 2026 evidence base

Start with these twelve sources

The full research audit reviewed 221 distinct 2026 sources and retained 205. These twelve cover the main claims in this tutorial. Dates below are the publication or update dates recorded during the 18 August 2026 review.

1Anthropic: Demystifying evals for AI agentsChecked 2026-01-09 2Anthropic: Test, measure and refine Agent SkillsChecked 2026-03-03 3AI Evals Course: Eval Skills repositoryChecked 2026-08 4LangChain: Run, trace and thread evaluationChecked 2026-06-23 5SWE-Skills-Bench: marginal utility of agent skillsChecked 2026-03-16 6SkillRouter: skill selection and rerankingChecked 2026-03-23 7MalSkillBench: runtime-verified malicious skillsChecked 2026-06 8LLM judges can be too generous without referencesChecked 2026-07-14 9SkillTV-Bench: skill-aware trajectory verificationChecked 2026-08-06 10Snyk: ToxicSkills security auditChecked 2026-02 11Amazon Science: TRAJECT-BenchChecked 2026 12AIES: LLM-as-Judge StandardChecked 2026-07-16

Research limit: community reports were used for failure discovery and practitioner context. Official documentation and research carried more weight for product and benchmark claims.

Build a local AI system that keeps improving

The local Second Brain path connects durable context, skills and review loops. Use this manual to test the skills before they become routine.

Explore the local AI setup

Keep decisions and failures available

MemoryOS gives your AI a durable memory layer. Pair it with regression cases so useful lessons survive beyond one chat.

Explore MemoryOS

Back to the Agent Skill Evals resource page