A skill is not a markdown file

A colleague sent me a skill they were proud of and asked me to take a look.

Four thousand words. A step-by-step walkthrough of how to authenticate against the REST API. A twelve-row table of endpoints that would be needed maybe once every fifty invocations. And an opening line I have now seen more times than I can count: “In order to accomplish this task, you will first need to understand the following concepts.”

It was written with the best intentions. It was making their agent worse. And all I had to offer was my opinion — which is exactly the problem.

That’s what agents-skill-eval is for.

Everyone is writing skills now

And honestly? Good. The idea behind agent skills is one of the better things to come out of the last two years. Put a capability in a folder — instructions, scripts, whatever the job needs — let the agent pull it in when the task actually calls for it, and stop re-explaining the same thing in every single prompt.

But look at the barrier to writing one. It’s a text editor. That’s the entire barrier.

So everyone is writing them. Engineers, product managers, marketers, people who have never once thought about what a context window is or what it costs. And almost every one of them shows up with the same mental model:

A skill is documentation for the AI.

More detail is better. More examples are better. Be thorough. Be complete. Leave nothing out.

That model is wrong. Not slightly wrong — wrong in the specific way that makes the thing you built do the opposite of what you wanted.

What “thorough” costs you

The Agent Skills specification is built on progressive disclosure. Three separate budgets, and you are spending from all three whether you know it or not:

• name and description — loaded at startup. Every request. Forever. Whether your skill is relevant to what you’re doing or not.
• The body of SKILL.md — loaded the moment the skill activates.
• Everything in references/ — loaded only when something genuinely needs it.

Most people write as if there is one budget. So:

• You write a rambling 300-word description, and you’ve just put a permanent tax on every request you will ever make in that runtime.
• You inline a forty-line bash pipeline, and you’re paying tokens to have the agent read something it could have simply executed from scripts/.
• You write “read all files in references/ before starting”, and congratulations — you have just defeated the entire reason references/ exists.

And a bloated skill is not even the biggest line on the bill. MCP is. Attaching a server drops its entire tool catalogue in front of the model before you’ve asked for anything — several thousand tokens as a floor, and routinely into the tens of thousands once you turn on a full toolset. The alternative, for most of that work, is a couple of sentences telling the agent which CLI to reach for. Call it 500 tokens.

Nobody wires up an MCP server maliciously. They wire it up because it’s easy and it’s the thing everyone is talking about. One config block, it works, it feels like progress. The cost never comes up.

And that’s how all of this compounds. The agent gets a little worse. A little more distracted. A little more likely to miss the one thing that actually mattered. And then everyone shrugs and says the model is having an off day.

It isn’t. You are.

The other failure mode: rebuilding what the agent already knows

Bloat is the obvious one. This one is sneakier, and I think it’s worse.

Someone I know wrote a skill called jira. It supported every operation you can imagine — create a ticket, update one, transition it to a status — and it exposed all of it through flags. A flag for the operation. A flag for the team name. A flag for the project. A flag for basically every field Jira has. By the time it was done it took something like fifty different parameters.

What he had built, in the end, was a wrapper around the Jira REST API.

Think about who that wrapper is for. No human is ever going to remember fifty flags — and no human was meant to, because the agent is the one calling it. So how did the agent do? Worse. Measurably worse. It couldn’t reliably work out which flags to pass in which combination, so it started making mistakes it wasn’t making before the skill existed.

And here’s the thing that should have stopped the whole project on day one: the agent already knew the Jira REST API. It is one of the most thoroughly documented APIs on the internet, and the model had read all of it. Nobody needed to re-teach it. Re-teaching it badly, through a homemade flag interface nobody else has ever seen, was strictly worse than saying nothing at all.

The skill wasn’t adding knowledge. It was adding indirection.

This is exactly why Tier 3 flags step-by-step tutorials on standard tooling, and why the MCP policy pushes you toward the CLI. Not because CLIs are cooler. Because if the model already knows the tool, every token you spend explaining it is a token spent making the model less sure of something it was already sure about.

Before you write a skill, ask the boring question: does the agent already know how to do this? If the answer is yes, the only thing worth writing down is the part that’s specific to you.

What a good one looks like, for a few dozen tokens

Here’s one that runs dozens of times a day, and it’s a good example because the useful data is hard to get at.

The difference from the fifty-flag wrapper matters, so it’s worth naming up front: that one re-exposed an entire API surface and handed the agent the job of driving it. This one collapses a single workflow into a single call with nothing to get wrong.

SonarQube runs on a pull request and posts a comment. That comment is a summary: a handful of new issues, coverage moved by some fraction of a percent, quality gate passed or failed. If you ask an agent to go fix the findings, that comment is close to worthless — it says something is wrong without saying where.

The detail is in the full report. Which file. Which line. Which rule fired. Whether this is a readability smell, an uncovered branch, a duplicated block, or a security hotspot. That’s what the agent needs, and it lives behind a paginated API, mixed in with every pre-existing issue in the project that has nothing to do with this pull request.

The tempting move is to write it all down — explain the API in SKILL.md, document the endpoints, describe the response shape, list the rule categories.

Scripts do it instead.

They call the API, page through the results, filter down to the files this pull request actually touched, and drop everything the agent has no use for.

So what does the agent do? It runs one script and waits.

The script handles the rest, including the retries, which is the part people miss. If a call fails, the script retries, and that costs nothing. Put that same retry loop in the agent and every attempt burns a full turn of reasoning and a pile of context. A retry is cheap in a script and expensive in an agent. Push it down.

And notice everything the agent never sees:

• Not the code in the script
• Not the auth, the headers, the pagination cursors
• Not the raw API envelope wrapped around every issue
• Not the hundreds of pre-existing findings that predate this pull request
• Not the failed attempts

What comes back is a compact piece of structured JSON — one entry per finding that belongs to this change. That’s the entire footprint in the context window.

And that shape isn’t what SonarQube returns. Every mapping is done in the script. Rule keys become categories the agent can act on. Severities get normalised into one vocabulary. Component identifiers get resolved into real file paths. Everything nobody needs gets dropped on the floor.

That work has to happen somewhere, and where you put it is the whole decision. Hand the agent the raw payload and it does the mapping — which means it reads every field to find the few that matter, reasons about what a rule key implies, and produces a slightly different interpretation each run. You pay tokens for the reading, tokens for the reasoning, and you get inconsistency for free.

Do it in the script and it costs nothing, forever. Same mapping every time. No interpretation, no drift, no tokens. The agent gets a clean list and spends its attention on the actual job — deciding what to fix and fixing it.

A few dozen tokens for the part that matters, instead of a full report the agent has to read, hold, and sift — while the thing you actually asked for slides further down its attention.

This is what progressive disclosure is actually for, and it’s the distinction I’d want anyone to take away from this post:

• SKILL.md is when and why — the judgement the model can’t have.
• scripts/ is how — code to be executed, never read.
• references/ is the detail, and it stays on disk until something asks for it.

The moment you find yourself explaining mechanics in prose, stop and ask whether that should have been a script. Deterministic work belongs in deterministic code. You don’t need a language model to page through an API and rename fields — you need it to look at what came back and decide what to do about it.

MCP isn’t banned. It’s priced.

The evaluator ships with an opinion about MCP, and I’d rather explain it here than have you discover it as a surprise finding.

Take GitHub. Its MCP server loads a tool catalogue running well into the tens of thousands of tokens before the agent has done anything at all. Yes, you can scope it down with toolsets — and you should — but even scoped, you’re paying up front for definitions the agent may never call. The gh CLI covers the same ground for a fraction of that, and the agent already knows gh, so you aren’t teaching it anything either way. Assume the most flattering number you like for the MCP side; it’s still a bad trade, which is why the argument survives being applied to almost every MCP server with a mature CLI behind it.

So two namespaces are blocked outright:

• mcp__github_ → use the gh CLI or the GitHub REST API
• mcp__atlassian_ → use the acli CLI or the Atlassian REST API

Two are explicitly allowed:

• mcp__figma_ — MCP is realistically the only way in
• mcp__slack_ — no good public CLI exists

Everything else is a warning, not an error. New namespaces get flagged for a human to classify rather than auto-condemned, because I don’t know your stack and the evaluator shouldn’t pretend it does.

That warning bucket is not a formality, and the sharpest example I have sits in it. I put the Wiz MCP server through a single ordinary question: what is the scope of this vulnerability — what does it affect, how bad is it, where is it. Getting to an answer that way consumed close to 100,000 tokens. The same question, answered by a handful of scripts against their API, costs a few hundred.

That is not a tuning problem. It’s two orders of magnitude, and it’s worth understanding where it comes from, because tool definitions are only the entry fee. The rest is the round trips. Every exploratory call returns a verbose payload, the agent reads all of it to find the part that matters, asks a follow-up, reads another one, and the transcript of working it out stays in the context for everything that comes after.

A script collapses that entire exchange. It knows which calls to make, makes them, and hands back the answer — not the record of arriving at it.

Wiz isn’t on my blocked list, and that’s deliberate: I’ve measured one workflow, at one company, at one point in time. That’s enough for me to write scripts instead. It isn’t enough to make the decision for you. What the warning does is tell you to go and measure it yourself, which is the whole point of the bucket.

Look at what separates the two lists, because it isn’t ideology. A namespace gets blocked when a perfectly good CLI already exists. Figma and Slack stay allowed precisely because they don’t have one. Banning MCP outright would be silly — plenty of services are MCP-only, and for those it’s the right tool.

The rule is narrower than “MCP bad”. It’s: don’t pay tens of thousands of tokens of tool definitions for something a 500-token CLI call already does. You get one context window. Spending a meaningful slice of it on definitions for tools the agent may never call in that session is a bad trade, and it’s a trade most people make without noticing they made it.

The detail I care about most is where this check lives. MCP findings sit in Tier 3 — token efficiency, not in the security tier. That’s deliberate. MCP isn’t dangerous. It’s expensive. Filing it under security would be dishonest and would give people the wrong reason to act on it.

And it’s a config, not a conviction. The lists live under mcp_policy in scoring_config.json, so if your org has an internal namespace that should always pass, or you disagree with me about GitHub, you change the config. You don’t fork the logic.

So stop reviewing this by hand

The premise of the whole project is boring on purpose: most of this is checkable without asking a model for its opinion.

The evaluator runs four deterministic tiers over a skill folder.

Tier 1 — Spec compliance. Frontmatter validity, naming rules, whether the directory name matches the name field, which fields the stable spec defines versus which ones somebody invented on the spot.

Tier 2 — Security. Tool scoping, destructive operations, hardcoded absolute paths that will only ever work on the laptop of the person who wrote it.

Tier 3 — Token efficiency. Inline code blocks over five lines. Reference tables over ten rows. Duplicated blocks above 80% similarity. Paragraphs that ramble past three sentences when one would do. Step-by-step tutorials on tooling the model already knows cold. Preload directives that break progressive disclosure. And the MCP namespace policy described above.

Tier 4 — Effectiveness. Ambiguity, missing examples, undefined defaults, operations that will happily do the wrong thing the second time they run.

Findings roll up into a 0–100 score and a tier: excellent, good, needs work, poor. Errors cost five points each, warnings two, and each side is capped so the arithmetic can’t run away. Three individual checks — a missing name, a missing description, and a skill that points elsewhere without saying anything actionable — cap the total on their own, so you can’t score well while failing the fundamentals.

There is an optional LLM review layer on top, weighted 40% against 60% deterministic. Its job is deliberately narrow: judge the false positives that regex will always produce, and catch the things that need reading comprehension. It does not re-run the checks. Leave the API keys unset and you get a pure deterministic run — reproducible, free, and fast enough to sit in a pre-commit hook.

Three ways to run it, because different loops need different things

The web app. Point it at a GitHub URL or upload a folder. For the person who wrote one skill and wants to know if it’s any good, without installing anything.

As a skill. The evaluator ships as a portable skill package. Drop it into your runtime and your agent can evaluate skills — including the ones it just wrote for you five seconds ago. That closes the loop tightest: written, evaluated and fixed before a human opens it at all.

In your pipeline, or on your own box. --ci emits structured JSON with a schema version, status, score, tier and grouped findings with reasoning. Fail the build under a threshold and the garbage stops reaching main. Or self-host the whole thing:

docker run -p 8080:8080 ghcr.io/sasa-fajkovic/agents-skill-eval-app:latest

Same evaluator behind all three. There is no hosted-only tier.

None of this is theoretical. These checks have been gating skills across an engineering organisation for months — in pipelines, and inside the agents writing the skills in the first place.

If you’ve read my post on unsupervised CI/CD, you already know where this is heading. A check that a machine can run consistently is a check that shouldn’t need a human in the loop at all. Skill quality is exactly that kind of check.

No vendor lock-in, and what that cost

This evaluates against the agentskills.io specification. Not against Claude Code. Not against Codex. Not against whichever harness your team happens to be running this quarter.

That constraint cost me features, and I knew it would. It would have been easier to validate the extra frontmatter fields individual runtimes support, and to score skills on how well they exploit one specific harness.

I didn’t, because think about what that produces. A skill that scores 95 on a vendor-specific evaluator and breaks the moment you switch runtimes hasn’t been evaluated. It’s been graded on how deeply it’s stuck.

We spent 2025 celebrating MCP as the USB-C moment for AI. Standardisation was the whole point. It would be a strange thing to accept a universal protocol for tools and then let skills — the actual knowledge, the part that’s genuinely yours — get welded to one vendor.

The same principle drove a decision that sounds arbitrary until you think about it: scripts/ should contain bash and Python 3. Nothing else.

Not because I dislike the alternatives. Because those two are the ones you can assume. Every CI runner, every Linux box, every Mac — bash and Python 3 are there or a standard step away, with no package manager, no version manager, no lockfile and no “works on my machine” conversation.

The moment a skill needs Node, or Go, or Deno, it quietly stops being portable and becomes portable conditional on a setup step. It now only runs where someone already installed that runtime. You’ve taken a folder that anyone could drop in and turned it into a dependency your teammate has to satisfy before they get any value out of it.

That is the same trap as vendor lock-in, one layer down. Lock-in isn’t only about which harness you’re on. It’s anything that makes your skill work here and not there.

So the evaluator grades on it. Check 1.11 lists the top level of scripts/, ignores data and config files, and sorts the rest into three buckets. .sh and .py pass. A recognised-but-discouraged runtime — .js, .ts, .go, .rb, .php, .pl — is a warning: you lose points, you don’t fail, and the trade-off is written down where your team can see it. An extension it doesn’t recognise as a scripting language at all is an error.

That middle bucket is the important one. If your whole team runs Node, a warning is the correct outcome — it tells you where your portability ends without pretending your working skill is broken. The extensions also get surfaced in metadata.unsupported_script_types, so a pipeline can make its own call without parsing findings.

Some honest caveats, since I’d rather say them than have you find them. The check reads file extensions, not shebangs — a .py pinned to a Python 2 interpreter sails straight through, and a .sh using bash 4 syntax passes on your Linux CI before breaking on a colleague’s Mac, which still ships bash 3.2. It only looks at the top level, so a runtime hiding in scripts/lib/ goes unnoticed. And an extensionless file — which is how plenty of shell scripts ship — isn’t classified at all. Extension-level portability is the floor, not the ceiling.

One distinction worth drawing, because someone will draw it for me: what gets graded is runtime-neutral, but the evaluator itself currently ships as a Claude Code skill. That’s where it happens to run today, not what it grades against. Rewriting the packaging for another runtime is an afternoon; rewriting criteria that were built around one vendor’s quirks is a rewrite.

The spec is the contract. If the spec grows, the evaluator follows. If a vendor ships something proprietary and useful, good for them — it still doesn’t belong in a portability check.

Is it perfect?

Of course not.

Regex will flag things that are fine. A five-line threshold on code blocks is a heuristic, not a law of nature, and sometimes your six-line block genuinely belongs inline. Tier 4 is the softest of the four, because “is this ambiguous?” is a harder question than “is this table sixteen rows long?” That’s precisely why the LLM layer exists, and precisely why it’s held to 40%.

More importantly: it never runs your skill. There are no evals against real agent transcripts behind these scores. It reads the package, not the behaviour. A skill can pass every check and still be the wrong tool for the job, and no amount of static analysis will catch that — measuring what a skill actually does to an agent’s output is a harder problem, and an honest one to leave open.

So it will never tell you whether your skill is useful. It can tell you it’s well-built. Whether anyone needed it in the first place is still on you.

But be pragmatic here. The alternative isn’t a perfect review — the alternative is no review at all, which is what’s happening right now. Optimise for the common case. The common case is a skill with four obvious problems that nobody looked for.

What I actually want from this

I want skill quality to become boring.

The same way linting became boring. The same way test coverage became boring. The same way formatting arguments got settled by a tool nobody had to agree with.

Right now, whether a skill is any good depends entirely on whether somebody senior enough to know what a context window costs happened to look at it. That doesn’t scale, and I doubt it’s happening at all — it’s hard to believe the number of people who could review this work kept pace with the number of people writing it.

A deterministic evaluator does scale. It runs in a pipeline. It runs locally. It runs inside the agent that’s generating the skill in the first place. It doesn’t need to be smart. It needs to be consistent, fast, and the same for everyone.

Try it at agents-skill-eval.com. The source is on GitHub.

Run your worst skill through it and see what falls out. The evaluator scores its own skill package 98 — and the one point it docks is a paragraph I wrote three sentences longer than it needed to be. Exactly the kind of thing you never catch by eye, and a checker never misses :)