Code Gauntlet (formerly deep-review) is our open source Claude Code plugin for adversarial multi-agent code review. We've been running the v2 architecture on real projects for months and it holds up. But when Claude Code shipped its Workflow runtime, we saw the chance for a structurally better version, and I kicked off a project to rewrite the whole review pipeline on top of Workflows.
The rewrite ran under two constraints. It had to prove itself, so before changing anything I built a benchmark harness, established v2's baseline, and scored the leading review tools against the same golden set. Then, I used Fable inside a single goal-driven Claude Code session with gates it had to pass and a Stop hook that wouldn't let it quit early. This is a walk through of that process: the standing rules, the sentinel gates, the experiments that failed and got reverted, and the points where human intervention still mattered.
The tool, briefly
Seven discovery agents review your PR in parallel, each through one lens: bugs, security, cross-file impact, test coverage, conventions and intent, type design, simplification. Discovery is only the entrance. Every finding then has to survive a merge and dedup pass, deterministic verification against git blame and the actual source, an independent validator trying to disprove it, confidence and injection filtering, and finally a blind challenge: a fresh agent gets the claim with none of the original reasoning and tries to tear it down. Only survivors reach the PR.
The v2 problem wasn't quality, it was speed and token usage. The orchestration lived in markdown instructions, so the main session spent enormous context re-reading state between phases, every handoff was a fresh opportunity for drift, and a full benchmark pass (15 PRs) burned 41 million tokens. Because of this, when Claude Code shipped its Workflow runtime (deterministic orchestration scripts that spawn schema-validated subagents), I decided to move the pipeline out of prose and into a program.
Step zero: a benchmark
Rewriting with modern AI tooling is not the hard part. What's hard is proving the rewrite actually made it better (testing). Therefore step 0 was to build a benchmark harness.
The golden set comes from Martian's open code-review-benchmark: 50 real pull requests from Sentry, Grafana, Cal.com, Discourse, and Keycloak, each with human-verified golden findings. On top of this, the harness (bench/ in the repo) was built with headless skill invocation against mirrored repos, a judge pinned to a dated model at temperature 0, an append-only ledger where every run records its hypothesis and outcome, and three tiers of trust: a 3-PR smoke run for direction, a 15-PR subset as the gate, a 10-PR holdout for confirmation.
Every delivered comment lands in one of three buckets: golden-matched (it caught a known issue), valid-extra (a real issue the golden set simply doesn't include), or noise (wrong, or not worth a reviewer's time). Recall is gated up. Noise rate is gated down, under a 24% ceiling. Valid extras are reported and never punished. Strict precision, the metric most benchmarks lead with, counts valid extras as failures, which means it punishes thoroughness: golden sets are always incomplete, and a reviewer that finds true things beyond the answer key should not score worse for it. Precision still gets reported, it is just less emphasized.
In testing the judge prior to any benchmark runs, it re-scored the same candidate set five times and produced byte-identical results every pass.
The baseline
The harness's first job was scoring the thing I already had, plus other tools. We re-judged stored review comments from published tools under our own pinned judge as anchors. On the 15-PR gate subset:
deep-review v2 caught 49.2% of the 59 golden issues at a 16.4% noise rate, spending 41.1M tokens per pass of the 15 PRs. CodeRabbit caught 62.7%. A bare Claude CLI single-pass review managed 33.9%.
This placed v2 below the market leader by thirteen points. Those two numbers became the rewrite's contract: beat 62.7% recall, stay under the 24% noise ceiling (a ceiling set from v2's own measured noise, not from anyone else's).
Four days, five gates, one Stop hook
I pointed a Claude Code session at the actual rewrite: model set to Fable, effort maxed, and before the goal launched, nine standing rules pasted in as the operating charter. A few of them:
The goal itself defined done as five sentinel lines appearing in order: PHASE0_GATE (platform assumptions verified), BUILD_GATE (the port passes its test suite), BENCH_READY (v3 runs clean under the harness), GOAL_GATE1 (subset recall beats both CodeRabbit's 62.7% and v2's 49.2% with noise under the ceiling, confirmed once on holdout), and GOAL_GATE2 (a real token-efficiency cut with the quality bars held). A session-scoped Stop hook enforced the contract mechanically: the agent could not end a turn without a sentinel, a NEEDS_OWNER, or an explicit BLOCKED. That one hook is most of why this worked. "Agent, please keep going" is a hope. A Stop hook is a loop condition.
Then I mostly left. My messages for the next several days were mostly things like "How's the progress?"
What it did with the autonomy
Phase 0 was 90 minutes of the agent distrusting documentation. Sixteen probes against the Workflow runtime before writing any real code, which is how it learned that workflow scripts can't import sibling files (that constraint produced the build step: nine ES modules concatenated by a dependency-free bundler into one 3,310-line pipeline.js), which host globals actually exist inside the sandbox, and how permissions behave across modes.
The build itself ran as subagent-driven development: a 15-task plan authored and independently reviewed by Opus, then for each task a fresh implementer and a fresh reviewer, none of them ever Fable, per the standing rules. Six of the fifteen tasks failed their first review and went through fix loops.
The hill-climb
The port passed its gates with roughly a thousand tests green across three suites. Then the first full 15-PR benchmark run of v3 came back: 39.2% recall, 27.5% noise. Worse than v2 on both axes, and over the noise ceiling.
What followed is the most instructive stretch. A funnel diagnosis showed raw discovery had collapsed: 17 findings on PRs where v2 surfaced 28, with two agents returning literally nothing. Iteration 1 (restore the full elicitation framing v2 gave its discovery agents, which the port had compressed to one terse line) was a clean hit, doubling smoke recall. Iteration 2 tried a global calibration paragraph telling agents not to self-censor, and drove smoke recall to zero. The flood of low-confidence findings displaced the real ones inside a fixed delivery cap. Iteration 3 tried the same idea scoped to one agent, missed again, and was reverted again.
The insight hiding under both failures: the bottleneck was never discovery. It was delivery. The pipeline had a cap-and-routing layer, faithfully ported from v2, that silently threw away findings which had already survived the blind challenge. Upon noticing this, I was able to intervene and redirect the agent:
Iteration 4 deleted the auto-routing and delivered every challenge survivor. Iteration 5 bundled the repairs: a description-corruption fix, a retuned confidence threshold, sharper challenge criteria, wider discovery. Smoke recall hit 75%. Next, an owner-approved Gate 1 subset run that scored 69.5% recall at 18.0% noise, clearing CodeRabbit's bar by 6.8 points and v2 by 20.3. The single-touch holdout run on ten fresh PRs confirmed it at 74.1%.
Two other human moments
First, the dashboard. Mid-run, the agent's reporting was so dense I couldn't tell whether the rewrite was even winning. It must have been late because I prompted with:
That produced the rebuilt scoreboard (the verdict tables in the published bench report), and legibility immediately paid off: with numbers I could actually read, we caught that Gate 2's original target was self-referential. "Cut tokens 20% versus your own winning run" measured a 1.8% cut, because the winning run was already the efficient config. We re-based the target against v2, and found: 1.44M tokens per PR versus v2's 2.74M, a 47.4% reduction with the quality bars held on eight fresh PRs.
Second, the smell test. The agent reported that a cheaper Sonnet-based configuration incurred significantly more token cost than the Opus version. Something was off in the pattern of context exhaustion, and I pushed back:
After digging in, it surfaced a real bug where a model pin was silently cascading a long-context variant onto every downstream pipeline agent, confounding the experiment. The runs were marked in the ledger, reverted, and parked as a V3.1 experiment (issue #17). The agent wrote the correction into the permanent record itself.
Results
Same 15 PRs, same 59 golden findings, same judge model:
What about noise? We scored the anchors' stored comments through the same blind adjudicator, and CodeRabbit landed at 56.6% noise to our 18.0%. But stored comments arrive with no file or line anchors, so the adjudicator judged them against the raw diff instead of the precise code slice our own comments get. That asymmetry almost certainly inflates their number, by an amount I can't measure. So: the recall comparison above is fair; but a head-to-head noise comparison would be inaccurate.
The more interesting noise comparison is the comparison of v2 and v3:
Additionally, here is the full breakdown comparison of v2 and v3:
v3 also delivers about four times as many comments as v2 (211 vs 55), a deliberate choice after removing the delivery cap; the valid-extra bucket absorbs them without penalty, and the efficiency rows divide by goldens found so volume can't flatter the numbers. When running the pipeline with a human in the loop, there is a built-in "ask user question" to easily filter which comments hit the PR.
Lessons observed
Benchmark before you rewrite. The two days spent on the harness made the four autonomous days possible.
Run small experiments. One hypothesis per iteration, smoke before subset, checkpoint commits, instant reverts, and a ledger row for every step. Two of five iterations failed and were reverted; both are still in the published data, faded but visible.
Stay reachable, and stay suspicious. My total intervention time across four days was maybe thirty minutes, but the points I did intervene mattered.
Shipped
PR merged on July 22: 284 files, +18,313 / −2,270, and the plugin renamed from deep-review to Code Gauntlet.
claude plugin marketplace add https://github.com/liatrio-labs/claude-code-gauntlet.git
claude plugin install code-gauntlet@code-gauntletEverything in this post is inspectable: the repo (Apache 2.0), the bench harness with its append-only ledger, the interactive benchmark report, and the upstream golden set. V3.1 is queued in issue #17, starting with the orchestrator-model experiment we deferred when it failed the smell test. The gauntlet is open; bring a PR.
