Five Scanners, 5,000+ AI Skills: What We Learned About Skill Security

AI skills are a major attack surface in agentic development. They are easy to distribute and install, and often run with elevated development privileges, making them a prime target for attackers. A skill bundles natural-language instructions, executable scripts, and access to external systems, making it both a force multiplier for the agent and a new avenue for manipulation if compromised.

That makes it critical to vet skills before they are used. Skill scanners aim to help by autonomously reading the whole bundle, surfacing what deserves a human's attention, and helping a team say yes to useful skills without having to hand-audit every one. Scanners play an important role as the first line of defense when evaluating whether to use a new public skill.

Fortunately, there are many skill scanners out there to help solve this problem. Unfortunately, there is no obvious best scanner, and most scanners can point to data that supports their own case.

Most scanners fall into two broad categories. Static scanners look for known patterns in the skill bundle: risky commands, suspicious URLs, encoded payloads, permission mismatches, and other signatures. They are fast, cheap, and deterministic, but they struggle when the risk depends on context or can be embedded in prose. LLM-backed scanners use a model to read the instructions and code together, which helps them catch semantic problems such as prompt injection, misleading descriptions, or behavior that does not match the skill's stated purpose. That extra context comes with tradeoffs: cost, latency, data-egress concerns, and probabilistic results.

We tested the tools ourselves, independently, with no horse in the race. Liatrio ran five skill scanners against publicly available corpora of malicious and benign skills and asked the practical questions: which tools produce signals worth acting on? Where's the tradeoff between false alarms and missed skills? When does stacking tools actually help?

The short version: LLM-backed scanners left the static-only tools far behind. Snyk gave the strongest single-tool verdict, and requiring three LLM scanners to agree produced the cleanest high-confidence result of anything we tried. Here is what we found.

How we built the benchmark

We tested the scanners against two separate tiers and scanned the full skill bundle, including SKILL.md and bundled scripts.

• Wild tier: We started with 597 body-bearing malicious entries from the public MalSkills corpus, then performed full-bundle fuzzy deduplication. That reduced the malicious set to 65 effective unique skills at a Jaccard threshold of 0.8. We paired those with 100 benign ClawHub skills, for a 165-skill run.

• Synthetic tier: We used MalSkillBench, a category-labeled synthetic corpus with benign facades and 15 behavior classes. To conserve cost, the LLM run used a subset of this corpus: a balanced 60-malicious, 60-benign set. The static run used 5,153 skills, making it the largest run in the study.

The two tiers answer different questions. The wild tier contains publicly disclosed malicious skills, but it is concentrated in a single disclosure stream from ClawHub and Koi Security. The synthetic tier gives broader, labeled behavior coverage, but it can carry generation artifacts. We report them separately rather than pooling them into a misleading single score.

The roster included:

We also ran three basic baselines: always flag, majority class, and a keyword heuristic. A meaningful scanner should clear those floors.

We picked one documented verdict rule per tool and applied it consistently within its mode, rather than retuning a threshold for every run. We measured five key metrics:

• Recall — of the genuinely malicious skills, what share did the tool catch?

• Precision — of the skills a tool flagged, what share were actually malicious?

• False-positive rate (FPR) — of the benign skills, what share got wrongly flagged?

• F1 — the harmonic mean of precision and recall: one number that rewards catching bad skills without burying reviewers in false alarms.

• Matthews correlation coefficient (MCC) — our main overall detector-quality score.

MCC folds all four cells of the confusion matrix — true positives, false positives, true negatives, and false negatives — into a single score from −1 to +1: +1 is a flawless detector, 0 is a coin flip, and −1 is confidently, perfectly wrong. MCC is unforgiving in exactly the way we needed: a tool that flags almost everything can post a healthy recall or F1 and still land near MCC 0, because indiscriminate flagging isn't detection — it's a rubber stamp. We also tracked cost per skill, measured with an in-guest man-in-the-middle proxy that logged token counts.

What the results say

Static scanning couldn't beat a keyword grep

Across both tiers, the best static scanner we tested never cleared the bar set by a generic keyword regex. As a standalone control, static scanning didn't earn its keep.

On the 5,153-skill synthetic run — our largest set — the top static scanners and the plain keyword baseline all clustered around MCC ≈ 0.34. In the table below, each cell is wild / synthetic (wild: 165 skills, 65 malicious; synthetic: 5,153 skills, 1,270 malicious).

Those values use the locked operating points in the primary analysis. We also tried retrospectively retuning each static scanner on the data to give it the highest possible MCC. SkillSpector improved the most, reaching 0.368 on the synthetic tier, but its wild MCC was still only 0.005. The broader conclusion holds: no static configuration was consistently ahead of the keyword baseline across both tiers.

Static tools did have one redeeming trait: specialization. skillscan-security was the only static tool with meaningful recall against the prompt-injection family — role hijacking, safety bypass, system-prompt leakage. AgentGuard was best on execution and persistence patterns like reverse shells and privilege escalation. If LLM-backed scanning is off the table, routing language-oriented findings through skillscan and executable payloads through AgentGuard beats trusting either as a complete verdict. It's a consolation prize, not a strategy.

And even that leaves holes. On the static synthetic run, 79 malicious skills — 6.2% of the malicious set — sailed past every static scanner. Static analysis can tell you a skill looks clean. It can't tell you a skill is clean.

Why signatures aren't enough

In theory, static checks should be the cheap win. Grep for dangerous commands, suspicious URLs, encoded strings, unexpected binaries, or permissions that don't match the job — fast, deterministic, no API bill. That's the pitch, but the data didn't agree.

We think this is largely because skills are so broad — arbitrary prose wrapped around arbitrary code doing arbitrary, mostly legitimate things — that pattern-matching tended to either fire on nearly everything or catch almost nothing, with little useful middle ground. The problem is semantic context. A scanner has to recognize when a friendly-sounding instruction is telling an agent to ignore its rules, when a remote installer has no business being in a commit formatter, or when a script's behavior contradicts the skill's own description. Those are comprehension tasks, not regex tasks. An LLM-backed scanner can read the prose and the code together and weigh both against what the skill claims to do.

LLM-backed scanning changed the picture

This is where the story turns. The LLM-backed scanners beat the keyword baseline on both LLM runs by a wide margin. The table below shows the locked operating points, wild / synthetic.

Snyk posted the highest single-tool MCC and the lowest false-positive rate in both LLM runs, but only when tuned correctly. That result depends on treating Snyk's E0xx malicious-pattern findings as a compromised-or-malicious verdict and its W0xx warnings as a separate posture signal. Count every warning as proof of malware, and Snyk lights up on ordinary skills that happen to use third-party content, URLs, or credentials. Read the codes the way Snyk means them, and it's the sharpest single tool in the study.

Cisco and SkillSpector led on recall. We ran both on Claude Sonnet 4.6. Cisco was the value option in this comparison, with roughly the same recall range as SkillSpector at about one-tenth of the cost. SkillSpector's cost distribution was much wider: its wild p99 was $7.54 for one skill, compared with Cisco's $0.16. These are not universal price claims, but they do reflect a general trend that SkillSpector is several times more expensive per skill than Cisco.

Results also need an error policy. Cisco had several unparseable or failed rows. On the wild LLM run, treating its failed malicious rows as misses reduced recall from 0.95 to 0.89. A deployment should record failures explicitly and avoid turning an unavailable verdict into a pass.

Combining scanners is an operating-point decision

The ensemble finding wasn't "more tools are better." It was that the decision rule — how you combine them — moves the tradeoff more than which tools you pick. In the table below, each value is wild / synthetic.

The three-way AND is an MCC-oriented recommendation for teams that need a manageable, high-confidence review queue. Different goals point to different configurations:

• Prioritize precision and reviewer capacity: Use the three-way AND to prioritize review. It cost about $0.58 per skill in our LLM runs, mostly from Cisco and SkillSpector.

• Prioritize recall: Use a union or majority rule, then expect more false positives and review them. Both combination methods flagged every labeled malicious synthetic skill in this run, but the union also flagged about 40% of benign skills. Switching to the majority rule drops the false-positive rate by more than half while holding synthetic recall at 1.00. The only cost is a little wild recall: 0.90 versus the union's 0.95. So the majority rule is usually the better recall-leaning choice, and the union is worth its extra false alarms only when missing that last wild skill is unacceptable.

• Optimize the synthetic tier alone: The SkillSpector AND Snyk pair had the best synthetic MCC, 0.915, slightly above the three-way rule's 0.899.

• Use one tool: Snyk's malicious-pattern verdict was the strongest single high-precision signal in this study, provided cloud analysis and its data-handling implications are acceptable. Cisco is the lower-cost, higher-recall alternative.

The right operating point depends on the consequences of a miss, the size of the review team, data-egress policy, and cost.

What a practical deployment should do

For teams using scanners as an autonomous first pass, the benchmark points to a few practical choices:

1. Inventory every file in a skill. Include scripts, dependencies, URLs, embedded configuration, and the tools or permissions the skill asks the agent to use.

2. Apply least privilege before scanning. A skill becomes more dangerous when the agent has broad access to a system. Don't allow permissions that aren't needed, and exercise extra caution when more access is provided.

3. Don't lean on static scanning. In our runs, it didn't beat a keyword grep. If you keep it, use it only to route findings by type as a cheap pre-filter — never as an approval decision, and never read an unflagged result as "safe."

4. Use LLM-backed scanning when policy permits. That is where the benchmark's useful signal came from. Snyk's malicious-pattern verdict was the best single high-precision signal in this study; Cisco and SkillSpector added stronger recall when their model use, vendor egress, reliability, and cost tails fit the environment.

5. Choose a decision rule deliberately. Three-way agreement produced the cleanest high-confidence queue. Majority or union rules broaden coverage, but they also create more false positives and review work. Match the rule to review capacity and tolerance for misses.

6. Keep human review for consequential skills. A skill that touches credentials, production systems, payment paths, or untrusted content deserves review even when no scanner flags it.

Limits of the findings

• This is a benchmark, not a universal ranking. The wild tier is narrow, the synthetic tier is generated, and both reflect the corpora available to us.

• Labels and runtime behavior have limits. We inherited upstream labels and did not execute malware, so label mistakes or runtime-only behavior may be missed.

• LLM results can move. Model versions, cloud tool behavior, public-corpus exposure, and API pricing can change. Treat large gaps as meaningful and small gaps as directional.

The takeaway

Treat AI skills like what they are: untrusted instruction-and-code packages that affect how an agent uses real permissions. The risks include disclosed overreach, external inputs that can be weaponized later, and behavior hidden in the full bundle.

The scanners that helped most were the ones that could autonomously read whole skill bundles and turn that review into useful triage for humans. On the corpora we tested, that mostly meant using LLM-backed tools: Snyk was strongest when we wanted a high-precision stop sign, while Cisco caught more suspicious skills at a fraction of the cost. The same pattern held for ensembles. A three-tool AND gave the cleanest high-confidence verdict across both LLM runs, while union or majority rules were better fits for teams that would rather over-flag than miss. Static scanning, by contrast, never made a case for itself as a verdict.

Scanners should not be the whole defense, but they are an integral part of one. Use them to vet and prioritize skills alongside least privilege, explicit approval boundaries, dependency review, and human judgment. Skills will keep changing, but the durable lesson is simpler: review the bundle, limit the blast radius, and let scanners help decide where humans should look hardest.

Sources and methodology

• MalSkills, CC-BY-4.0. Associated paper: arXiv:2603.27204.

• MalSkillBench repository and paper. The paper and repository state different reuse terms; this post reports benchmark results and does not redistribute the corpus.