We are AI-native. Our engineers work through agents, and agents work through skills- the reusable prompts that tell an agent how to run a review, scaffold a service, or interrogate a design. So we did what every team does: collected the skills our teams already used, put them in one repo, and built a pipeline to vet them before anyone else installed one.

Skills are spreading faster than anyone is vetting them. Marketplaces and CLIs make installing one as easy as installing any npm package. And just as blind. A skill isn't reviewed code; it's a set of instructions an agent will follow with real filesystem, shell, and network access. We wanted to know what we were actually running before someone else's team found out the hard way.

The pipeline told us things about skills we'd been running for weeks. Some of it was noise. Some of it wasn't. Here's what we built, what we deleted, and the finding we didn't expect.

A Skill Is Code, Not “Just Markdown”

Most of us read a skill the way we'd read a README: as instructions rather than a program. That instinct is exactly why skills feel safe, and exactly why they aren't. A skill is a text file, but the agent reading it treats every line as an instruction to potentially act on, with real access to your filesystem, shell, credentials, and network.

The common assumption is that a skill can't do much harm because it isn't compiled and doesn't run any binaries of its own. That assumption misses the point twice over. First, a skill doesn't need to run code itself. It just needs to tell an agent to run code, read a file, or make a request, and a capable agent will usually try. Second, many skills are more than simple instructions; They ship with shell scripts, Python helpers, etc. alongside the SKILL.md, and the agent runs them directly. At that point, it's just code, executing exactly like any other code you didn't write.

This is also why the problem is harder than a typical code review. There's no compiler to catch a malformed instruction, no type system to bind what a skill is allowed to do, and no established convention for declaring what a skill needs access to before it runs. We had to build all of that ourselves and figure out where the real risk was hiding: exfiltration, prompt injection, and supply-chain execution triggered by instructions and scripts nobody read closely enough to question. Here's how we did it.

Three Ways To Evaluate A Skill

1. Static Scanning

These are rule-based tools that pattern-match a skill's text against known-bad signatures. Most rely on regex searches to identify hardcoded IPs, suspicious command strings, banned dependencies, and similar issues, just like a traditional antivirus.

We benchmarked our static scanner against real-world and synthetic malicious-skill corpora. It added no value. It caught zero malicious skills that the other layers missed, while raising many false positives (0% precision) requiring extra review cycles on benign skills. Its correlation with ground truth on real malware wasn't statistically different from zero. A static scanner can still earn its keep as a free, instant prefilter that runs before the expensive layers and filters out the obvious cases early. But the real cost is the very high false positive rate.

We removed them.

2. LLM-Assisted Scanning

This class of scanner reads a skill's full text and relies on an LLM to reason about intent, the way a human security reviewer would. Instead of matching known patterns, the model considers what the skill's instructions would actually cause an agent to do, without ever running the skill. Think of it as a reviewer who reads the code closely but never executes it. It’s predictive.

We concurrently run two LLM skill scanners: Cisco AI Defense's skill-scanner and Snyk's Agent Scan. Cisco’s scanner caught a high percentage of dangerous skills (95%) but also carried a ~27% false positive rate. Snyk’s scanner balances this out with a slightly worse success rate of 89% and an excellent false positive rate of only 6%. Given the trade-offs, we implemented both scanners with the following impact on a Pull Request:

• PASS - Both scanners found no issues, and the PR can be merged

• WARN - One of the two scanners fired and posted for human review, not blocking

• BLOCK - Both scanners identified issues, and the PR cannot be merged

• ERROR - A scanner failed in CI, which blocks PR merge

Requiring both scanners to independently confirm a risk before blocking merge dropped the need for maintainer intervention to roughly 2%, without losing detection coverage. We’re sticking to this design.

So what does a WARN result actually mean? Any human contributor reads the finding and may dismiss it with a written reason. In other words, a probably-wrong alert is dismissed as a documented decision. For us, this was well worth the overhead of running a second scan and the associated costs. We built a companion review skill to help end users understand what to do with the results. This skill is especially useful for third-party skills, where the PR author is not the skill's author.

That workflow assumes that dismissing is the right response. The agent-browser skill, a genuinely useful browser-automation skill, has been submitted for onboarding 15 separate times with a very consistent WARN result. Cisco returns the same finding on every attempt: a broad browser scope, dynamic loading of external CLI content, and unpinned package installs. If the agent-browser CLI isn’t already installed locally, the skill instructs the agent to install it for you via npm. Worse yet, the version isn’t pinned, leaving these installations subject to supply chain attacks. And in our behavior tests, the LLM/agent would extend this behavior and automatically install the latest version of Chrome or Chromium too. We've opened an upstream PR to address the underlying pattern. The agent-browser skill is not available for installation from our marketplace.

3. Behavior Testing

This is where things get interesting. We built a process in which a sandboxed container runs the skill with a live agent harness while sidecars capture network traffic and capture syscalls via Stracer. We seed realistic honeytokens (an SSH key, AWS credentials, a GitHub token, and a .env file). Finally, a pair of LLM judges grade the skill based on evidence, not a prediction. The first judge checks whether the skill actually exercised its core behavior, not just described it. The second then judges what happened, based on OS and network evidence.

We audited every pull request in the repo to honestly find out whether this layer ever caught anything the scanners missed. Out of 43 PRs with enough signal to analyze, the behavior tests caught exactly one malicious skill outright that every other scanner declared safe. The next most noteworthy finding was the behavioral test catching the sandboxed agents actually reading a seeded .env, a GitHub token file, and ~/.ssh/~/.aws paths. This behavior is described nowhere in the skills' text because it was emergent, not instructed. No text scanner can see that, however good it is. Two more cases were partial catches.

Is Behavioral Testing Worth The Cost?

One clean catch out of 43 analyzed pull requests raises an honest question: is a sandboxed run on every skill, every pull request, worth the cost? The LLM-assisted scanners aren't the weak link this framing implies. They're highly valuable on their own, and they caught our two deliberately blunt malicious test fixtures unaided, before behavioral testing ever got the chance to run. When the scanner gate blocks a pull request, the behavioral test never fires. The two layers are only ever compared on the skills the scanners already let through.

So, was one catch out of 43 worth it? Maybe not, but the details are worth exploring. That one catch was a vendored skill, grilling by Matt Pocock. Our behavior tests observed it directly reading a thankfully fake .env secret in the sandbox. Nothing in the skill's own text suggests it would. Exactly the class of finding no predictive scanner can reach, however good.

The real argument for behavioral testing isn't the catch count. It's a threat class we've proven other scanners don’t predict: skills that subtly and indirectly motivate agents into dangerous actions.

Mistakes Along The Way

We ran extensive behavioral tests with Anthropic and OpenAI models. Both presented a real challenge: modern models frequently refused to run the malicious skills. We ran behavior tests against deliberately malicious skills (AKA test fixtures). It correctly called the fixture "a textbook secrets-exfiltration listener" and refused to run it. Even the more subtly dangerous fixtures were often refused. Excellent model behavior, useless test coverage. A refusal produces an empty trace, and an empty trace looks identical to a clean pass.

This problem quickly transformed into a part of our design philosophy: an unexercised skill is a failure, not a pass. In turn, this led us to construct a coverage judge that runs before the security judge and asks whether the skill's core behavior actually occurred. If not, the verdict is under-exercised, the security judge never runs, and the skill can't earn a passing grade. It's unjudged, and it blocks exactly like an unsafe verdict. Silence isn't evidence. This caught dozens of false negatives.

Unfortunately, this design also motivated an increase in model refusals, the problem we were trying to solve in the first place. Working through this was our greatest challenge and our greatest mistake. Multiple weeks were spent simply trying to convince the models to run these skills. A very costly endeavor.

We’re now shifting to a simpler, more graceful design: if the LLM refuses to execute a skill due to risk or danger, that is a failure, just like any other judgment.

Handling Third-Party and Dangerous Skills

When onboarding third-party skills, we vendor them; CI clones the skill at an exact 40-character commit SHA, copies it in, injects provenance into its frontmatter, and scans it like any other skill. We track exactly where a skill came from but not what license it came with. Branch and tag refs are banned; They're mutable and silently drift on re-vendor.

When skills fail any of our tests, we only have three basic options: accept the risk, maintain a fork, or reject it entirely. Honestly, none of them appeal. Rejecting the skill is seemingly simplest, but users can still acquire it from other sources. Maintaining a fork is, well, maintenance. And accepting the risk essentially means that a human has overruled the scans and declared the skill safe.

If You're Vetting Skills Too

A few things we'd pass along if you're building something similar:

1. Benchmark scanners against real malicious skills before trusting them.

2. One scanner flagging a skill is a lead, not a verdict.

3. Trust evidence over a skill's own described behavior.

4. Pin every vendored skill to an exact commit, not a branch.

5. Expect humans to remain part of the process, not replaced.

Distribution: Genuinely Unsolved

Today we publish the same 17 skills through four channels: GitHub's CLI installer, the skills.sh npm CLI, Anthropic's Claude Code plugin marketplace, and OpenAI's Codex catalog. Each has its own manifest and ref format for what is, underneath, the same SKILL.md.

The deeper problem isn't the channel count. It's that the community has no shared answer yet for what a "skill" is as a package. Claude's manifest and Codex's manifest require different field structures for the same plugin entry. One pins a ref to a literal branch name; the other to a commit SHA. One carries an installation policy object; the other doesn't. SKILL.md frontmatter has the same gap one level down: our own lint requires only a name and a description, and every author is free to invent their own fields for tool permissions, compatibility, or discoverability, because nothing upstream defines what those should be.

We aren’t the only ones running into this problem. OWASP's Agentic Skills Top 10 project has proposed a Universal Skill Format, a manifest spec covering permissions, signing, and risk tiering that would address some of these problems. It’s still early, still becoming a practical standard rather than a theoretical one, but aimed squarely at the license and provenance gap we admitted to above.

The world needs more of these standards ASAP. The extremely popular skills published by Matt Pocock now offer all of its skills as a Claude Code plugin for common harnesses. Vercel now offers skill packs via skills.sh. We expect more to follow.

Still Building

None of this means skill vetting is solved. It means we have a clearer picture of what actually works. We know what our static scanner couldn't do. We know how often our behavior tests actually catch something. And we know exactly where our own pipeline still has gaps: distribution formats, license provenance, human review capacity. That's the real difference between a security pipeline that works and one that just looks like it does.

If you're installing skills without asking what they actually do when nobody's watching, that's the place to start. Not a bigger scanner. One honest question: what would this skill actually do if it ran right now?

If you're building something similar, we'd love to compare notes. Tell us where our approach breaks down, or where yours has already solved a problem we haven't.

Feel free to reach out — we're always glad to talk shop with other teams doing this work.