Leaning Into Uncertainty on Code Reviews
I set out to replace Gemini Code Assist with local AI models. The end result is a new darkmux mission system, and a code review AI crew that extends our code quality gate in an unexpected way.
The review that flagged its own fix
darkmux runs a local model as an automated PR reviewer: no cloud API, nothing leaves the machine, self-hosted end to end. The first real test of that idea was a pull request fixing a malformed-credential bug that hung an authentication endpoint. I pointed an 8B local model at the diff. It came back with a near-clean pass, one high-severity finding, and it was the bug that was fixed in the diff. Then Gemini Code Assist reviewed the same diff, caught three real issues, and left a comment noting the local review’s quality. Both reviews sat on the same pull request, side by side.
That’s not an embarrassing footnote. That’s the actual starting condition of this whole project: a model confidently restating a solved problem as an open one, on a diff where the fix was right there in front of it.
Some context for why this matters professionally. My day job is CTO of a fintech startup, and the code under review is the kind where mistakes compound: billing math, subscription charges, lender-facing APIs. A bug that survives review doesn’t file a ticket on itself; it mischarges a customer. Every pull request gets an automated review from Gemini Code Assist, and it has earned that seat, routinely catching real issues that human review and tests both missed.
Two constraints made this worth two weeks of measurement instead of a shrug. First, that reviewer is going away: the consumer version of Gemini Code Assist’s automated PR reviews shuts down on July 17, 2026 (deprecated a month earlier, on June 18), and every team leaning on the free tier needs a replacement or accepts a hole in their review process. Second, the replacement has to run on the machine we actually have: an always-on Mac Studio with 32 GB of unified memory, which caps a loaded model at roughly 14 to 18 GB. That rules out the 70B-and-up class entirely. If a self-hosted code reviewer was going to work here, it had to work with small models: an 8B, a 24B, at most a 35B mixture-of-experts. Those are the stakes, and if you’re reading this because your own cloud code reviewer is sunsetting, they’re probably your stakes too.
One more thing before the story starts, because “two weeks” undersells it. This was not two weeks of letting models cook while I watched something on a second monitor.
What the lab did: the server logs from those thirteen days count 3,592 chat completions and 11.3 million tokens through the local models. Eleven different local models were auditioned, swapped, and re-tested, plus four cloud model configurations and the Gemini control. The same six pull requests got reviewed hundreds of times over, under different prompts, contracts, context windows, and reasoning budgets, with the busiest single day pushing 1,291 model calls through one Mac.
What it took to drive it: Claude Code ran the program with me, on a mix of Opus 4.8, Sonnet 5, and Fable at varying effort levels, the heavier configurations on planning and judgment, cheaper ones on execution, with Sonnet subagents doing the build swarm. Its session transcript logs over fifteen thousand assistant turns, six thousand tool calls, and 27 million output tokens in the same window, not counting that subagent swarm. My side of that conversation was roughly thirteen hundred messages: picking the next experiment, catching the bad inference, naming the variable that mattered, killing the run that was answering the wrong question. That division of labor, an operator steering an orchestrator steering a lab, is itself part of what this article is about.
The obvious question about that first review is why. The less obvious question, the one that ate the two weeks and produced most of what follows, is what actually fixes it.
If you want the thesis of this piece in one artifact, go read PR #1491: darkmux reviewed 2.0’s own code on local models, flagged a real bug, and the comment thread explains why it was right to merge anyway. It explains itself better than I can here.
Two weeks of fixes, and the wall that didn’t move
The first round of fixes worked, and each one shipped as a darkmux feature the same week (#1053). Give the reviewer the author’s own PR description and the false positives about the fix itself vanish: the shipped 8B reviewer (qwen3-8b at the time) went from two high-severity findings restating the fixed bug to a clean, correct summary of the same diff, because now it knew the change was a fix and not a bug. Give it the team’s coding standards as an injected resource and it recovers findings it can’t reach from the diff alone (a missing type annotation is a team convention, not a diff-visible fact). Fix the output schema, which had exactly one per-line channel and every channel was labeled “finding,” so a model with nothing wrong to report had nowhere to put “this is fine” except a manufactured problem. Each fix was tested in isolation against the same anchor case, one variable at a time, in darkmux’s lab harness, before any of them merged.
None of it closed the actual gap. Once the easy wins were spent, I built the measuring stick I should have built first: a labeled corpus of six real pull requests from my day job, each with a known bug and a fix-trail proving it, scored against Gemini Code Assist as the control while that control was still available. The first reading of that ruler, on a reviewer sized to actually fit a real machine, was zero. Zero bugs caught out of six, across three separate runs. Not a crash, not a degenerate output: clean “pass” verdicts on diffs carrying a critical double-charge bug and a date-range bug that silently dropped its last day.
I spent real effort making sure that zero meant something. A tool-call benchmark I built alongside the review bench, seeded fresh every run with nonce tokens so nothing could be memorized, showed the same model driving tools cleanly: seven of eight mechanical axes at pass-cubed (three clean runs each), zero fabricated answers across two dozen trials, six-hop tool chains held without dropping the thread. The model can use a computer. It just can’t review code with it. That gap, near-perfect hands against zero review recall, on the same model in the same afternoon, is the first real measurement in this whole story: not a vibe about small models being bad at review, a number with a control condition.
I ran the same six-case corpus against four model families (a 35B local reasoner, a frontier model with reasoning off, the same model with reasoning turned up, and two additional cloud models) across strict and freeform output contracts. Every diff-only cell, every model, every contract: recall pinned near zero.
DIFF-ONLY REVIEW RECALL, SIX LABELED PRs, 3 RUNS EACH
──────────────────────────────────────────────────────
model contract recall
──────────────────────────────────────────────────────
local 35B (reasons natively) strict JSON 0/18
local 35B (reasons natively) freeform 1/18
frontier (reasoning off) strict JSON 0/18
frontier (reasoning off) freeform 0/18
frontier (reasoning medium) freeform 2/18
──────────────────────────────────────────────────────Give the frontier model read access to the actual repository instead of just the diff, and recall moves for the first time in the whole grid, to roughly 28 percent (5 of 18 trials across three runs). Give the local model the same repo access, and something stranger happens: it drowns most of the time, burning hundreds of thousands of tokens wandering the codebase and producing nothing parseable, but once, in one of three runs, it caught the date-range bug that no frontier configuration, at any reasoning budget, on any model family tested, had ever caught. One time in three. The metered models are billed into discipline: they confirm what they can already half-see and stop. The model with nothing metering it wanders, and mostly that’s wasted, and once it was the whole job.
The clean read on all of this: diff-only review isn’t a model-quality problem. It’s an evidence-boundary problem. A diff review is a witness statement from someone who wasn’t at the scene, and no amount of thinking harder fixes a witness who wasn’t there.
The cap that was suppressing the answer
Access closed part of the gap. It didn’t close all of it, and the part that stayed shut turned into the most useful accident of the whole project.
I was watching the local model’s reasoning stream live on the dashboard when I noticed a pattern: the model kept saying, in effect, “I’m done, but wait, let me check one more thing,” working through something real each time, and then stopping mid-sentence. Not because it had finished. A per-turn token cap I’d set months earlier, calibrated against an older generation of non-reasoning coders, was cutting the model off right as it closed in on the answer.
I scraped the trajectories to check. Across recent dispatches, 43 to 50 percent of turns were hitting that cap mid-reasoning. One turn, 51,000 characters of reasoning on a real bug, opened by tracing the actual method where the defect lived, named the exact issue in progress (”a potential issue here: the origin parameter...”), and got cut off one tool call short of confirming it. The final words before the cut: “Let me check the actual implementation of...” Then nothing. That’s not a hang. That’s productive investigation, interrupted mid-sentence.
The naive fix is to raise the cap. I didn’t, because the published literature on reasoning-model behavior is explicit that more room is not free: models can self-entangle in redundant loops the longer they’re allowed to run, and correctness doesn’t track token count in a straight line. So instead of picking a bigger number, I removed the cap entirely, to see what the model actually does unleashed.
The result was half of what I expected, and the wrong half was the interesting one. Some turns did exactly what I’d hoped: the model reasoned its full length, finished the thought, and produced a real finding. Other turns, given no ceiling at all, never stopped: fifty thousand tokens deep and still circling, until the case timeout killed the run and it produced nothing. Same model, same task, and the two shapes are indistinguishable by length while they’re happening. The runaway begins exactly like the deep, productive turn does. There is no number, not ten thousand tokens, not a hundred thousand, that saves the first without dooming the second. The cap was never too low or too high. It was measuring the wrong axis.
And then the proof that mattered more than the cap itself: on the single hardest bug in the corpus, the model given all the reasoning room it wanted and the whole repository to read anchored on the exact function, named the exact fix, and used one specific word seventy-four times across twenty-two thousand characters of thought. It had every piece. It held them for the length of a short story. And it flagged two things that weren’t wrong and missed the one that was.
Eureka: the machine wasn’t thinking too little. It was thinking about the right things and unable to conclude. The leap from “here are the facts” to “therefore this is broken” is a different faculty than the one that generates the facts, and giving the first faculty unlimited room does nothing for the second.
That reframes the whole project. Fixing the cap matters, it’s a real harness bug and it’s worth shipping correctly, but it was never the recall lever. More reasoning length was never going to bridge the gap between holding the facts and generating the accusation.
Lean into uncertainty
Every dead end up to this point had the same shape once I looked back at it: a contract that demanded certainty the evidence didn’t contain. A binary pass/fail verdict pushed a model with a genuinely fifty-fifty read toward the safe answer, which is a rubber stamp. A “don’t flag what you can’t prove” instruction pushed recall to zero and kept it there, because on real code, static proof of a lot of real bugs doesn’t exist: how a value is populated at runtime, whether a caller ever passes null, what happens under load. An explicit “decline if you’re not sure” escape hatch didn’t produce more honest declines; under a defect prior, it produced confident fabrication instead, because declining wasn’t a respectable output and the model would rather invent certainty than sit with the discomfort of having none. Give the model unbounded time to reach certainty and it investigates for the better part of an hour and files nothing.
I ended up giving the shape a name: the absolutist’s paradox. The harder a contract demands certainty from a stochastic mind, the less honest work it gets back; the demand doesn’t produce rigor, it produces one of three counterfeits (silence, a rubber stamp, or invented confidence). It’s the best rule of thumb I know for context engineering now. When you write a prompt, a schema, or an agent contract, find every place it implicitly says “be certain,” and ask what an honest mind that ISN’T certain is supposed to do there. If the contract gives that state no legitimate output, you’ve already chosen which counterfeit you’ll receive.
The training-scale version of this is now established literature: OpenAI’s “Why Language Models Hallucinate” (later in Nature) showed that binary accuracy grading rewards guessing over abstention, so models learn confident error at the benchmark level. What I kept running into is the same law operating one level down, in the individual contract: you don’t need a leaderboard to teach a model to bluff, one prompt that refuses to accept “I’m not sure” does it in a single call.
The lesson underneath all of it: lean into the uncertainty instead of fighting it. Every honest epistemic state a model can actually be in, confirmed, worth a second look, or wrong, needs a legitimate output that isn’t punished. Punish miscalibration, being confident and wrong, never punish an honest hedge. A senior human reviewer’s “something smells here, I can’t prove it yet” is high-value output. What destroys trust isn’t the hedge. It’s confidence that doesn’t pay off.
The corollary I didn’t expect: allowing that uncertainty isn’t just more honest, it’s dramatically cheaper. A reviewer instructed to only file what it could prove spent twenty to forty-five minutes per case scouring the codebase, burned reasoning turns tens of thousands of tokens long chasing closure, and often filed nothing anyway. A reviewer licensed to flag on suspicion, “maybe something is wrong here, here’s why,” took nine to seventeen seconds on the same code with the same model. That’s roughly a hundred and fifty times faster, on the same weights, because the only thing that changed was the proof burden. Demanding proof of something that’s statically unknowable is an unbounded search by construction. You can search forever for evidence that doesn’t exist inside the one file you’re holding.
So: don’t ask one pass to be sensitive and disciplined at the same time. Split the job. A cheap first pass gets full license to be wrong and gets rewarded for speed and coverage. A second pass, with a different mandate and different evidence, gets rewarded for tearing the first pass apart. Collapse to a verdict late, at the stage that can actually measure, not early, at the stage that can only guess.
The breakthrough: make the small model’s job easier
Before the funnel could be built, one more thing had to break, and it was a belief of mine, not a benchmark. I had been staring at the competition wrong. Gemini’s reviews were not, I eventually understood, a giant model heroically reading a diff. They were almost certainly the visible tip of a pipeline: heavy static analysis run up front, a deep book of pull-request history consulted, and the model narrating conclusions the machinery had already staged. The AI might just be the narrator.
Once I saw it that way, the question changed from “which model can review code?” to “how much of reviewing is not a model’s job at all?” The answer became a procedural bundler that runs before any model wakes up: callee bodies fetched, parameter flows traced, sibling functions surfaced, unknown symbols declared as explicitly out of evidence, excerpt truncations marked instead of hidden. All deterministic. All free. The small model stopped being asked to be a reviewer and started being handed a bounded question it could actually win. Nothing in the two weeks moved the numbers more than that shift, and no part of it required a better model. The measured version of the same lesson had been sitting in the grid data already: no contract, effort level, or model tier rescued diff-only review, but giving the model the code’s surroundings did.
Building the funnel: a prosecutor, a judge, and a verifier
The design that fell out of the two lessons (lean into uncertainty and pre-digest the diff for the model) has three seats.
The prosecutor runs first, cheap and fast, with an explicit strong prior that a defect exists somewhere in the code it’s looking at. It doesn’t need to be right. It needs to be sensitive, and it needs to be fast enough to run wide: many small, cheap probes over many bundles of code, rather than one long agentic wander through a whole repository. A 4B model given a well-scoped bundle of the changed function plus the sibling code it needed to compare against reproduced a frontier reviewer’s hardest catch of the whole corpus in nine seconds. That’s not a fluke of a good day; it’s the direct payoff of removing the proof burden. Under a strong defect prior, a small model is a fine detector. It is also, predictably, a fine fabricator on code that has nothing wrong with it, and that’s fine, because catching the fabrications isn’t the prosecutor’s job.
The judge is the seat that catches them. I designed the judge as the author’s agent, not a neutral arbiter and not an adversary of the code: its incentive is that a false alarm wastes the author’s time and a missed defect ships under the author’s name, so both error types cost the same principal. That symmetry matters. A neutral judge has no stakes and drifts toward whichever answer sounds most defensible; an adversarial “find every flaw” judge over-blocks. An author’s-agent judge is trying to protect one specific person from both kinds of mistake, and that’s the incentive structure that gets calibration instead of theater. The judge’s contract has three outcomes: confirmed, needs a second look, or false positive, and the design principle behind that third option is the same lean-into-uncertainty rule from before: every honest state gets a legitimate answer, and only miscalibration gets punished.
Beyond the judge sits the author’s own coding agent, a verifier that can actually compile the code and run the tests.1 That’s the top of what I’ve started calling the verification-power gradient: a probe makes a pattern judgment on facts it’s handed, a judge checks a claim against the bundle it has, and a coding agent checks a claim against reality. Kill every flag at the cheapest stage that can actually settle it. A hedge that survives the judge and lands in front of a coding agent isn’t a failure of the funnel, it’s correct routing: that flag closes as easily as it arrived, for the cost of one test run instead of an hour of a model trying to simulate what a compiler already knows.
The design choice underneath all three seats got its own confirmation at the smallest possible scale. I ran a single bundle of code through a range of prompts, from a strong defect prior down to a soft, calibrated one, on two different models. The result was a straight diagonal: whichever model held onto its catch also kept fabricating on clean code, and whichever model learned restraint lost the catch. One model argued a real defect was intentional, in two seconds, once its prior was softened enough to consider that possibility. No single setting, on either model, bought both recall and restraint in the same dispatch. The seesaw that killed every earlier single-pass prompt held even down at the level of one bundle, one model, one shot. That’s the strongest evidence I have that this isn’t a prompting problem with a clever answer waiting to be found: recall and restraint are two different attractors in the same weights, and you don’t get to stand in both at once. You have to build two seats.
Five laws
Measuring the funnel at scale produced five findings that behave like laws: they held every time I looked for a counterexample, across models, across bug classes, across weeks.
The register law: confidence carries zero grounding signal
I gave the prosecutor a graded output contract instead of a flat one: report a finding as PINNED (I can explain exactly why this is wrong) or as SUSPICION (something here reads wrong, I can’t fully explain why). The prediction was that real bugs would come back PINNED and weak or invented findings would degrade to the honest, lower-confidence SUSPICION. Across seven test cells, including three that had nothing wrong with them, the prosecutor chose PINNED seven times out of seven. It never once chose the honest hedge, even on code it was actively fabricating a defect against.
Eureka: choosing between “certain” and “unsure” is a self-report of grounding, and grounding is exactly the signal a small model doesn’t have access to. Retrieval and confabulation produce identically fluent, identically confident prose. Humility requires knowing what you don’t know, and that requires the verification step the prosecutor specifically isn’t doing, which is what makes it cheap in the first place.
That closed a question I’d been circling for weeks: don’t ask the prosecutor to self-report how sure it is. Assign the confidence grade downstream, at the seat where it can actually be checked against the code.
The repair-distance law: verification reaches an adjacent hypothesis, and no further
I gave the judge one extra option beyond kill-or-confirm: if a flag’s stated reasoning was wrong but the flagged code had a real problem nearby, propose the corrected mechanism instead of just discarding the flag. On flags where the correction was a small step, a slightly wrong theory about the same underlying mechanism, the judge converted two of two cleanly, including reasoning its way to the actual labeled bug from an unrelated starting theory. On flags where the correction required an entirely different, unrelated theory, the judge converted zero of three: it correctly ruled out the stated reasoning and then simply stopped, never surfacing the real defect sitting in the same bundle it had just examined, even when it had confirmed that exact defect from a different flag on the identical code minutes earlier.
Verification is not generation. A model checking a claim against code can walk a short distance from that claim toward the truth. It cannot originate a fresh accusation once the one it started with is dead. That’s the same wall from the reasoning-cap chapter, showing up one seat downstream: more room to look doesn’t grant the ability to originate a new idea, and neither does more room to verify.
The evidence-scope law: every mind is exactly as right as its evidence
A live event handed this one to me for free. Gemini Code Assist filed a real, high-priority false positive on a real production PR: a claim that two properties of an object were drifting out of sync. The fact that refutes the claim, that the two properties are actually one storage slot with an alias, lives in a different repository than the code under review. I replayed the exact claim through my own local judge at two evidence scopes: the scope Gemini actually had, and the scope with the refuting fact added in.
At Gemini’s scope, my local judge independently confirmed Gemini’s exact wrong verdict. A second frontier model, given the same repo-scoped evidence and no knowledge of Gemini’s answer, reconstructed the identical wrong mechanism from scratch, with a tighter argument. The moment the refuting fact entered the bundle, the same local judge flipped to a decisive false-positive ruling, citing the exact mechanism, in forty-four seconds.
The law that falls out: at fixed evidence, every model I tested, from a model small enough to run on a phone to the frontier, was exactly as right as the evidence in front of it, no more and no less. “Which model is smarter” dissolves as the interesting question. What the harness assembles into view is the system’s intelligence. The false positive was manufactured by an evidence boundary, and it was cured by moving the boundary, not by upgrading any mind on either side of it.
The seat law: the judge’s quality is the funnel’s precision
On the same clean test case, code with nothing genuinely wrong with it, a 35B local judge returned zero confirmed findings. Swap that judge for a 4B model, same case, same prosecutor flags feeding it: ten confirmed fabrications. Zero percent precision on the identical input, from swapping one seat.
The judge is the highest-volume seat in the funnel: every flag from every prosecutor passes through it (after deduplication). Whatever the judge’s actual discrimination ability is, that’s the ceiling on the whole pipeline’s precision, no matter how good the prosecutor was upstream. A weak judge doesn’t just underperform. It turns the funnel into a noise amplifier with an extra step.
The mandate law: same model, different assignment, disjoint defect classes
This one showed up a level above code review, in the swarm that built the funnel itself, and it’s worth its own section below.
The numbers
The full funnel, run once end to end against the day-job corpus: 528 flags in, from every prosecutor bundle across every case.
THE FUNNEL, ONE FULL SWEEP: 528 FLAGS IN
────────────────────────────────────────
outcome count share
────────────────────────────────────────
false positive 470 89%
confirmed 29 5%
needs a second look 29 5%
────────────────────────────────────────
items an author actually 58 11%
readsRecall through the funnel held at 100 percent retention against every known bug the prosecutor fleet had ever caught: nothing that should have survived to the confirmed tier got lost in the kill. A separate frontier verification pass, three independent read-only agents auditing the confirmed and needs-a-second-look tiers against the actual repositories, put a harder number on precision: three of the five diff-visible labeled bugs confirmed as real, plus two real bugs the funnel found that weren’t in the original labels at all, against a confirmed-tier precision of roughly 61 percent (about eleven real findings against seven false alarms, out of eighteen items adjudicated so far).
That’s worth putting next to an external number instead of grading it against nothing. DoorDash published a similar production architecture, a cheap scout followed by a stronger reviewer, the same generate-then-filter shape as this funnel, and reported it lifted their weighted recall from 30.7 to 53.6 percent at flat precision, with their deployed frontier system landing at roughly 53.6 percent recall and 87 percent precision. That recalibrates the whole project’s ambition: the bar was never “match or beat the frontier’s single-pass number,” which nobody in production is actually hitting either. The honest deliverable is the structure delta: the same weights doing meaningfully better work once they’re split across seats with different jobs, which is exactly what a solo local reviewer sitting at roughly 11 percent recall and a funnel-fed local pipeline clearing 40 to 60 percent on the same corpus demonstrates.
Recall inside the funnel isn’t a fixed number either, it’s a draw probability, and that turned out to matter more than any single prompt change. Running the same bundle five times against known bugs split the corpus into three distinct classes:
RECALL BY DRAW COUNT, THREE BUG CLASSES
─────────────────────────────────────────
bug class @1 @5 @10
─────────────────────────────────────────
surface-salient 100% 100% 100%
mid-difficulty 20-25% ~76% ~94%
needs external fact 0% 0% 0%
─────────────────────────────────────────Some bugs are caught by every draw, so running the prosecution probe more than once buys nothing on that class. Some bugs are a genuine sampling problem, roughly a one-in-four to one-in-five shot per draw, and running five draws instead of one turns that into a three-in-four catch and ten draws into better than nineteen-in-twenty, for the cost of a few extra seconds per draw. And some bugs don’t respond to draw count at all, because no amount of resampling manufactures a fact the model was never shown; that class needed the actual sibling code in the bundle, not more attempts at the same evidence.
Raising the reasoning temperature to try to buy more diversity on the stuck class did the opposite of what I hoped: it produced zero additional recall and a wider spread of different, equally wrong fabrications on clean code, which is strictly worse for the judge downstream. Temperature is not a coverage knob here. Draw count is.
A model bake-off against the stuck class settled the last open question about it. Two bug cases had gone 0-for-13 and 0-for-10 respectively against every local model I’d tried through the entire project. A 27-billion- parameter reasoning model, still small enough to run on a laptop, cracked both: three for three on one, and the first hit of the entire project on the other, after a single ten-minute draw. A mid-sized non-reasoning model was the best instrument for the mid-difficulty class instead, and a 9B reasoning model scored zero everywhere. Reasoning ability alone wasn’t the ingredient. Reasoning ability at enough scale was, and 9B sat below that threshold on this task.
The last case I’d been calling unsolvable, the date-range bug that dropped its final day, the one no frontier configuration at any reasoning budget had ever named, fell to the combination: put the sibling function’s actual code in the bundle, not just its name, and run the 27B reasoner against it. Roughly two catches in three draws, versus zero in three when the sibling code was absent (the model instead converged, three for three, on a different, real, previously unflagged issue in the same function). With every lever now measured on its own axis, sibling-code inclusion, reasoning scale, draw count, the honest projection for a hybrid team run to completion is a recall union in the 85 to 90 percent range against the diff-visible labeled bugs, up from a solo ceiling of roughly zero to eleven percent when this project started.
What certainty costs
The wall-clock story underneath all of this is as sharp as the recall story. The reviewer instructed to only file what it could prove spent twenty to forty-five minutes per case and often produced nothing. The reviewer licensed to flag on suspicion took nine to seventeen seconds on the same code, same model, roughly a hundred and fifty times faster. The funnel’s own docket sweep, roughly 530 flags through the judge at about 25 seconds per judgment with zero API errors across more than 180 consecutive calls, ran to completion in under four hours, serialized, on a laptop.
Certainty, it turns out, is the actual cost driver, not thinking, not model size. The tiered design is a market for certainty: spend it at the stage where it’s cheapest. A probe pattern-matches and hedges in seconds. A judge checks a claim against the code it’s handed and hedges the rest in tens of seconds. A coding agent that can compile and run tests settles the remaining hedges for the cost of one test run instead of an hour of a model trying to simulate what a compiler already knows for free. This tracks with a broader finding in the self-verification literature: when a model rechecks its own reasoning inside the same context, the recheck comes back overwhelmingly confirmatory rather than corrective. Real correction needs a different reader with different evidence, which is a fair description of what the prosecutor-to-judge handoff actually is: not a recheck, a rethink. And we weren’t alone in converging on the split: DoorDash’s production AI reviewer arrived at the same architecture at fleet scale, a scout that notices feeding deep reviewers that verify, with an explicit precision-over-recall posture. Different company, different stack, same shape.
The swarm that built the funnel
Phase B of this project, the piece that turned the design into a working darkmux pipeline, shipped in a single day, built by roughly fifteen agents working in parallel under frontier orchestration: seven feature packets, five coverage passes, two small fixes, about fourteen pull requests total, several individual agents running past three hundred thousand tokens of their own work. Every PR passed through two review tiers before merge. A mid-flight design change (a rename that touched several packets at once) got absorbed without stopping the line. One agent got killed by a session limit partway through and picked back up with zero lost work.
The result that mattered most, though, wasn’t the speed. It was what different agents caught, given the same underlying model but a different assignment. Implementers building strictly to spec missed doctrine violations that weren’t in the spec to begin with. Reviewers reading each diff against the spec caught protocol and honesty gaps the implementers couldn’t have seen, including a vacuous-pass bug in the funnel’s own judge logic: a code path where “the judge cleared everything” and “the judge never actually ran” produced the same green result. A third wave of agents, assigned specifically to audit merged code against the project’s own documented promises rather than against a spec or a diff, caught what both earlier waves missed: a forward-compatibility break, a flag that was silently being dropped instead of honored, and a write path handling real data with zero direct test coverage.
Nobody in that swarm was smarter than anybody else. It was the same model, dispatched with different mandates, and each mandate aimed the model’s attention at a different contract surface. That’s the mandate law from the funnel itself, one level up: an implementer, a reviewer, and a coverage auditor looking at the same code will find near-disjoint sets of problems, not because one of them is more capable, but because they’re reading for different things.
The thing underneath all of this that I keep coming back to: a multi-agent system built the multi-agent review system, and the review system’s own laws, evidence boundaries determine verdicts, convergent independent findings are a confidence signal, diversity of assignment beats redundancy of capability, kept showing up as properties of the swarm that built it.
The loop closes
The clearest proof that any of this was worth building came within minutes of the first real sweep finishing. Three of the funnel’s frontier-verified confirmed findings became backlog tickets, described with the funnel’s own reasoning and verification chain attached. One of them, a billing-math bug caused by an unpinned timezone default (the exact bonus catch the cloud control had missed entirely), got picked up by a coding agent and moved into a fix the same day. Raw flags in, a fix in flight, same afternoon, with a full evidence chain behind every step in between.
The follow-up audit on that one bug was the sharpest validation of the whole approach. Tracing the same defect class across the rest of the codebase surfaced nine separate sites carrying the identical unpinned-timezone pattern, feeding two different jobs that write revenue-bearing records on a schedule. The existing test suite couldn’t have caught the family at all: the one test that exercised the relevant code pinned the date but never varied the timezone. That’s worth sitting with for a second, because it cuts against a tempting conclusion: the prosecutor’s strong defect prior had been generating dozens of false timezone claims all afternoon, killed correctly by the judge every time, and the one genuine timezone bug in the whole corpus turned out to be the tip of a real, systemic problem in the codebase. The prior wasn’t wrong about the codebase. It was wrong about most individual sites and right about the class. Suppressing that prior to cut down on false alarms would have buried the day’s single most valuable finding along with the noise.
The out-of-sample test I had planned, a held-out pull request the tuning had never touched, stopped being a lab exercise before I could run it cleanly. The deadline arrived, and the funnel went to work on real pull requests it had never seen during any tuning decision. Production became the out-of-sample case.
The deadline arrives
Gemini Code Assist’s automated reviews went dark on schedule. The replacement had to be standing when they did, and it was, but not in the shape the project started in.
The 32 GB Mac hit a wall first. The full funnel, run at production cadence on real pull requests, maxed the box: load-and-unload thrash between seats, an occasional unbounded model load that could hang a whole review, no budget-aware scheduling to pack the models that had to be co-resident. Models that fit the machine may still need tuning before they run the whole funnel reliably there. So the seat that had been a local judge became, for the cutover, a hosted GPT-5.1 endpoint. Not a triumphant “we added cloud seats.” A retreat under fire that kept reviews flowing while the local path gets hardened.
That retreat is a better lesson than any capability boast would have been. You will hit hardware walls on the machine you own. The thing that lets you survive them gracefully, decide when to spend a frontier seat, watch what the box is actually doing, catch the slowdown before it becomes a mystery, is observability, not a bigger model. It’s exactly why darkmux is growing lab tooling and host telemetry into the viewer. The frontier capability is a footnote; the hardware wall and the instrumentation it forced are the paragraph.
And it left the funnel in the shape it should have been in all along: the seats are venue-flexible. A prosecutor, a judge, a verifier, each one is a slot that can be filled by a local model on your hardware or a hosted frontier endpoint over the network, chosen per seat as a config decision rather than a rewrite. The dispatch runtime stopped caring where a seat’s brain lived; it needed only to know how to reach it and how to authenticate. Tier-fluid by design.
The cutover surfaced one more thing I hadn’t expected, and it sharpened the case for owning the loop. Gemini auto-reviewed pull requests posted by a human account and refused bot-authored ones; even an explicit trigger comment from a bot account wouldn’t start it. In an engineering process where a bot posts the PRs, which is increasingly the norm as agents write more of the code, a reviewer with that policy covers a shrinking, human-only slice. Agent-authored code is exactly where automated review matters most, and it was exactly the code the cloud reviewer wouldn’t touch. The funnel runs on workflow dispatch, indifferent to who authored the diff. Taking control of the loop was never only about cost or the sunset. It was about covering the work your own agents produce.
Different, not better
With both reviewers running on the same production pull requests, every PR that carried both reviews became a clean head-to-head: same diff, same moment. Once Gemini’s preview window closed for good and the comparison corpus froze, I read it finding by finding, mechanism by mechanism. I went in expecting a scoreboard. I got a species map.
Start with the thing I had been muttering all along: the hosted GPT-5.1 reviews felt verbose. The data agrees, and it took two passes to say so honestly. The first cut of the numbers said the opposite, that the funnel was leaner on most pull requests, until I noticed the corpus was mixing eras: the tiny reviews from the pre-funnel local-model months were dragging the median down. Cleaned to the honest GPT-5.1-era subset, the perception was correct. The funnel posts more text than Gemini on two-thirds of the matched pull requests, median around 8,000 characters against roughly 2,600, about three times as much, and the tail is violent: the top five funnel reviews hold 67 percent of all the review text it posted, with the worst ballooning to seventy-eight thousand characters against Gemini’s ceiling under nine thousand. So, corrected in public: the first read was wrong, the gut was right, and the fix was cleaning the corpus, not softening the claim.
Where the length comes from is worth naming precisely, because it’s not what it looks like. Eighty to ninety percent of a large review’s text is the model’s own per-finding prose, and the driver is findings count, not per-finding bloat. GPT-5.1 confirms about 6.6 findings per PR to Gemini’s 2.1 posted inline, and per finding it runs a median 2,196 characters to Gemini’s 1,476, longer but nowhere near threefold. The threefold gap in total text is the threefold gap in findings surfaced. Verbosity is the price of coverage, not waste per finding.
And the part of the cost you can’t see at all: GPT-5.1 is a reasoning model, and reasoning tokens are billed as output while staying invisible in the posted comment and, it turned out, in darkmux’s own telemetry. A review that posts a short comment can have burned heavy reasoning to get there, on top of every probe, judge, and verify generation the funnel runs internally and never posts. The visible review is the tip. The bill is the iceberg, and a monitoring tool that cannot see reasoning tokens is measuring the wrong half. That’s a filed fix now, and it’s the same observability-at-home lesson this project keeps relearning.
Then the finding that reframes everything: the two reviewers agree on almost nothing. Matched at the defect level, across the union of everything both of them found, the overlap is about five percent. Even at the coarser file level they land on the same file only fifteen times out of sixty-odd distinct file touches. They’re frequently reading different files in the same pull request. Neither is close to a superset of the other.
That five percent isn’t noise. It’s the structural signature of two different context windows, and once you see it that way the comparison clicks into place. The funnel reads one bundle at a time: a changed function plus the sibling code it imports, deep and narrow. Gemini reads the whole PR at once, wide and shallow. So the funnel is a depth-first reader and owns the defects that require tracing one function’s full execution: data-integrity and logic-correctness bugs at 38 percent of its findings against Gemini’s 20 percent, the wrong-index selections and silent overwrites and commit-before-confirm hazards, plus an entire API-contract and HTTP-semantics lane, 8 percent of its findings, that Gemini never entered at all. Gemini is a breadth-first reader and owns the defects you catch by seeing every file at once: surface null and type-safety issues at 33 percent against the funnel’s 20 percent, spread uniformly across the diff, plus performance, plus test-versus-implementation drift, a species the funnel is structurally blind to because its bundles never include the test files.
On the thing I most wanted to know, whether the local funnel finds as many high-stakes issues as the cloud control, the answer is yes, modestly more: roughly 15 to 20 distinct critical or high defects to Gemini’s 13. Not a scoreboard win. The funnel and the cloud reviewer surface comparable quantities of serious issues and then barely overlap on which ones. Two lenses, same diff, and the small shared region is exactly where the top-severity items cluster: a timezone day-shift both caught independently, a protected field leaking through a merge that both flagged from different directions. That last one is the quiet result the whole project was pointed at. A local prosecutor and judge, on a 32 GB Mac, independently reconstructing a cross-boundary security-relevant finding a frontier reviewer also caught. Not more, not sharper. There, arrived at from a different direction, which is what makes the agreement a confidence signal instead of a coincidence.
I kept the method honest by killing a hypothesis I liked. I had assumed cross-file contract breaks, the class where a change in one file quietly violates an assumption in another, would be Gemini’s home turf, the payoff of its whole-PR context. In the strong form, that died against the data. The corpus had few pure cross-file-contract defects, and Gemini’s whole-PR edge showed up as breadth, null-safety everywhere and style and perf, not as deep cross-file logic. The one clean whole-PR-only win it kept was test-versus-implementation alignment, and the funnel loses that one for a boring structural reason, not a reasoning one: it never reads the test files. The blind spot is a scope decision, not a capability gap.
The ground-truth number under all of this comes from an independent adjudication pass on the funnel’s findings, resolving each one against what actually got fixed: about 65 percent true positives, about 35 percent not-an-issue. That 35 percent is the honest cost of the depth-first read, the false-alarm tax you pay to find the logic bugs the wide pass walks past. Gemini has no equivalent false-positive ground truth in this corpus, so I won’t pretend to a matching number for it. What the two numbers I do trust say together is this: on real, distinct, actioned defects, the two reviewers are comparable, the funnel modestly ahead on volume, and almost entirely non-overlapping in what they see.
So the claim is not “better.” It is “different.” The design is the difference: context-window shape predicts the species skew, and the five percent overlap is its fingerprint.
The general recipe
Every finding here rhymes with every other, and that’s not an accident of a tidy write-up. It’s what kept showing up when I went looking for it. A probe fleet works because different models construct different theories about the same code. The funnel works because different evidence scopes produce different verdicts about the same claim. The swarm that built the funnel worked because different mandates read the same code through different lenses. At every layer, diversity of perspective beat redundancy of capability, and variance, once I stopped fighting it and started measuring it, turned out to be a feature: two independent readers converging on the same finding is a confidence signal no single reading can produce on its own. The head-to-head made that concrete one final time, at the level of whole vendors: two reviewers with opposite context windows, agreeing on almost nothing, and the narrow band where they did agree was the part you could trust most.
The funnel was never a special trick for PR review. It’s one instance of the general recipe for reliable work from stochastic minds, and it was built by another instance of it.
Keep both
The project started with a subtitle: replacing Gemini Code Assist. It succeeded by failing. You cannot replace an instrument with one that shares five percent of its readings; the measurement that was supposed to crown a winner described a partnership instead.
So we kept both. The pipeline runs on hardware we already own, tireless and private and tuned to the depth-first logic bugs that mischarge a customer. Gemini Code Assist stays on, in its paid edition now that the free preview has ended, for the breadth we’re structurally blind to, the null-safety and the test drift and the wide-angle pass that catches what no single bundle contains. Together they cover what neither sees alone. Paying for one and running the other beside it isn’t hedging. It’s the honest read of two instruments that measured out to be complementary by construction.
That’s a more useful answer than a winner would have been. A team that knows it’s holding two different lenses aims them differently than a team that thinks it swapped one for an identical spare. And one of those lenses has an aperture ring. Gemini’s evidence scope is fixed and opaque: whatever its pipeline decides a pull request is, that’s what it reads. The funnel’s evidence is an input you compose. Hand it the whole PR, or one commit, or a single file you’re suspicious of, or a diff with the vendored noise cut out before any model spends a token on it. The evidence-scope law says every mind is exactly as right as its evidence; owning the workflow means the evidence is a dial you set, not a decision a product makes for you.
Which leaves the questions open, the only honest way to end a piece named for leaning into uncertainty. Can a local system built on the machine you already own stand alongside the big-tech offerings, sometimes matching what a company with a planet’s worth of indexed code behind it produces? (Google’s code intelligence traces back through Kythe to an internal project literally named Grok, years before that word meant anything else, which tells you the scale of the moat.) Sometimes, yes, and it’s worth being precise about what “sometimes” is up against. Can the whole thing be made fluid enough that every seat, prosecutor and judge and verifier, is a choice you make per stage: local, or Azure, or Google, or whoever posts the best number for that job this month? That part is built. Can it grow a visual lab that shows you where your own sweet spot is, on your own hardware, with your own code, and lets you decide for yourself instead of taking my word for it? Both ship as darkmux 2.0: a mission orchestrator and a lab, every seat staffed local or cloud, with the review funnel you just read about as its flagship mission.
None of that is a conclusion. It is a stance. Embrace the uncertainty, keep the operator in the loop at every seat, and value observability over authority. That is darkmux culture, and the story it’s telling has just begun.
Appendix: the concept index
Every named idea in this article, one line each, with the research it converges with and where the working implementation lives. The code is open source: darkmux (the funnel arc is issue #1222; the observability arc is #1247).
The absolutist’s paradox. Demanding certainty from a stochastic mind returns counterfeits: silence, a rubber stamp, or invented confidence. Give every honest epistemic state a legitimate output; punish only confidence that doesn’t pay off. Training-scale convergence: Why Language Models Hallucinate and its Nature version.
Make the small model’s job easier. Most of “reviewing” is not a model’s job: static analysis, context assembly, and evidence gathering are deterministic and free. Hand the model a bounded question. The implementation is the funnel’s procedural bundler.
The register law. A flag’s confidence markers carry zero signal about whether it’s real; branch on verifiable acts, never on a model’s epistemic self-assessment.
The evidence-scope law. A judge can only rule on evidence inside the bundle; symbols referenced but not defined there must be declared, or the boundary itself manufactures false positives.
The repair-distance law. A judge can strengthen an adjacent hypothesis but cannot generate a fresh one; if the prosecution never states the right mechanism, no amount of judging recovers it.
The seat law. The judge seat takes the call volume, so staff it with a throughput model (MoE-class); slow deep reasoners belong in prosecution seats with bounded dockets.
The mandate law. The same model under a different assignment produces a disjoint defect class; coverage comes from varying the mandate, not just the model. Related self-verification evidence: The Self-Verification Dilemma, which measured the same confirmatory-recheck failure this law routes around.
Double confirmation. Re-judging only the confirms with an independent pass intercepts unstable verdicts at the cost of one extra ruling each; disagreement demotes rather than deletes.
The see-saw. Oscillating one knob between two bad poles (strict contract kills recall; loose contract floods flags) feels like iteration but is the signature of a missing dimension; the exit is an architectural split, not a better position on the axis.
Salience is not a defect map. Wide models’ flags cluster on interesting-looking code, not defective code; in our data the fixed code drew 13 flags telling one story while the real bug drew 3. Coherent high-volume clusters mark smoke, not fire.
Early exits don’t survive verification. Every rule for stopping a deep prosecutor early (fixed finding counts, consecutive-dry-draw windows) died against verified data: in our measurements the one real catch arrived near the END of the draw order, behind a parade of confident false confirms that would have satisfied any early-exit trigger. Deep seats run their full selected docket; spend the savings budget on structural bundle selection instead.
What certainty costs. Chasing closure burns reasoning tokens without adding recall; convergent with the overthinking literature (surveyed in arXiv 2508.02120) and with DoorDash’s production review architecture, whose scout-then-verify split carries the same precision-over-recall posture.
Venue-flexible seats. A seat (prosecutor, judge, verifier) is a slot, not a model; each can be filled by a local model or a hosted frontier endpoint as a per-stage config choice. The cutover to a hosted GPT-5.1 judge under memory pressure was a retreat that hardened into a design feature, not a capability boast.
Different, not better (the species map). Two reviewers with opposite context windows overlap on only about 5 percent of the union of their findings, because context-window shape determines defect species: a bundle-scoped depth-first reader owns logic/data-integrity and HTTP-contract defects, a whole-PR breadth-first reader owns null-safety, tests, perf, and style. Comparable critical/high yield, near-disjoint coverage. The design is the difference, so the honest conclusion is complementarity, not replacement.
Posted text is the tip; the bill is the iceberg. On a reasoning model billed per output token, the visible review understates the spend: reasoning tokens are invisible in the posted comment and were absent from darkmux’s own telemetry, and the funnel generates far more (probe, judge, verify) than it ever posts. A monitoring tool that cannot see reasoning tokens is measuring the wrong half.
Earlier work in this series: the Genesis articles cover the darkmux lab methodology this program ran on.
The design names one more seat, not built on this run: a defender that argues a flag is wrong before the judge rules, giving the accused code its own advocate. A natural extension of the author’s-agent judge, but not yet worth the extra dispatch on the current corpus.