The local model won one of my five tools.Copy link

Two days after Qwen3.8-Flash-Next was released, it was serving on a DGX Spark on my network. The day after that, it beat Claude Opus 4.7 outright on one of the five production tools in my benchmark. Not on a leaderboard, not on a demo set. On my own tooling, my own corpus, scored by the same pinned frontier judge as every previous run.

It also lost to every model I've ever put through the harness on another of the five. The loss tells me as much as the win does, so it gets the same space here.

The setup, brieflyCopy link

For anyone who hasn't read the earlier pieces in this series, I keep a 52-case benchmark across five internal CLI tools I actually run: code-assess (find the seeded bug in a diff), extract-structurise (freeform text into a schema), test-strategy (propose target behaviours for a module), change-describe (commit descriptions from diffs), and log-triage (diagnose a 200-line window of real production logs). Every model gets the same corpus, and every judged score comes from Claude Opus 4.7 on an exact pin, with the model labels stripped before grading.

One bit of hygiene worth calling out, because it's the reason I trust the comparison at all. Before scoring the new model against anything, I re-ran the judge over the previous two arms to prove the old scores hadn't drifted. They moved by 0.02 to 0.04 F1, inside the noise band earlier passes had already documented. Same corpus, same judge, stability re-proven, then compare. Anything less and you can't tell a real change from judge noise.

The model, and the trick that made it fit on the SparkCopy link

The architecture is why any of this runs on desktop hardware. Flash-Next is 125 billion parameters as a mixture-of-experts, with only 6 billion active per token, plus a 51 billion parameter n-gram lookup table. The NVFP4-quantised checkpoint is 122GiB on disk. The box is a DGX Spark with 128GB of unified memory, no separate VRAM, drawing desktop power on my own network.

On paper that doesn't fit alongside everything else the box does. The trick, which comes from the blazux/qwen3.8-Flash-DGX community recipe (itself built on vLLM's own Flash-Next release branch), is that the lookup table doesn't need to be resident. Memory-map the 44GiB table from NVMe instead of loading it, and the resident weights drop to roughly 76GiB. With speculative decoding it serves at 28.5 tokens a second on structured output (JSON, code, the predictable stuff the draft model guesses well) and about 22 on free prose, where far fewer draft tokens get accepted. The structured number is the one that matters for tool calls, and it's comfortably usable; the old 27B shows the same split at roughly 24.5 and 15.

Getting there was not smooth. The serving recipe crash-looped six times at 93% loaded, with nothing useful in the logs, because the memory-map patch wanted a local snapshot path and I'd handed it a model id. Separately, the box's memory accounting overshoots its configured budget by a constant 30GB or so, which means you plan against measured numbers, not the flags you set. Neither of those is in a model card. Between them they cost me a couple of hours.

Where it wonCopy link

test-strategy is the tool where the line got crossed. Flash-Next scored F1 0.791. My previous local model, Qwen3.8-27B, sits at 0.674 on the same corpus and judge. Claude Opus 4.7 sits at 0.625.

The composition tells you what actually improved. Precision jumped to 0.749, against 0.552 for the old local model and 0.459 for Opus. Recall gave up a little (0.849 vs 0.906 and 1.005), so the precision gain wasn't bought by dropping coverage. And it was better on precision in nine of the ten cases against the old local model, so this isn't one outlier dragging an average.

Precision is the axis that matters most in a test-planning tool. A strategy with too many candidate cases is noise, not help.. well, worse than noise, because every low-value proposal is review time a human spends saying no. The frontier model's near-perfect recall comes from proposing nearly everything, and the judge marks it down for exactly that.

The economics land on the same side. Flash-Next ran this tool 37% faster than the old local model (53.6s vs 85.5s per call) on 26% fewer completion tokens. Opus costs roughly $0.102 per call here, the most expensive of the five tools. The local call costs electricity, on hardware that was already on the desk.

Honesty about the sample: ten cases is not a large corpus. The direction is unambiguous, the point estimate carries genuine uncertainty, and the judge is an Opus model grading a field that includes an Opus model, blinded labels or not. I'd want the corpus doubled before calling this anything stronger than a routing decision. It is, though, a clean win on real work, and a year ago I'd have laughed at the suggestion.

Where it lostCopy link

log-triage is the other half of the story, and it gets equal attention because it's the same model, the same week, the same harness.

Root-cause classification dropped to 0.63, against 0.81 for the old local model and 0.94 for Opus 4.7. Anomaly plausibility 3.63 vs 4.25 and 4.75. Next-checks usefulness 3.50 vs 3.75 and 4.38. Worse than everything, on all three judged axes. And it was 26% slower than the old local model on top (38.5s vs 30.5s per call), the only tool where it loses on wall-time as well as quality.

The failure mode is worth a look. Schema validity held at 100%, so the tool never broke. The envelopes are well-formed; the diagnosis inside them is weaker. And the regression concentrates on the longest, multi-hour log windows in the corpus, which fits what you'd expect from a model with 6 billion active parameters trying to hold a long operational narrative together. It writes a confident, valid, structurally perfect triage of the wrong problem.

That last sentence is why I don't publish the win without the loss. A model that beats a frontier model on one tool and produces plausible wrong diagnoses on another is not a model you promote estate-wide on the strength of a headline number.

The middle threeCopy link

The remaining tools fill in the picture. On code-assess every model scored a perfect F1 with zero false positives on the no-bug control, so the deciding axis is latency, where Flash-Next is 1.9x faster than the old local model (10.7s vs 20.6s per call). Seeded synthetic bugs, mind, not real regressions, so parity here is parity on an easy corpus. On extract-structurise it matched the old local model on every quality axis (0.89 schema validity, 0.56 exact match, 0.84 field recall), down to failing the same case in the same way. And on change-describe both local models sit within 0.09 of each other while Opus holds a wide lead on every axis (4.83 / 4.58 / 3.83 against Flash-Next's 4.00 / 3.33 / 2.83), so anything customer-visible stays frontier-routed.

Route the tool, not the estateCopy link

Here's what I take from the whole pass. Capability is now jagged enough that a 6-billion-active-parameter model beats a frontier model on one production tool and loses to everything on another, in the same estate, in the same week. Neither "local models have caught up" nor "local models are toys" survives contact with the per-tool numbers. Both are estate-level claims, and the estate is the wrong unit.

So my routing table now reads per tool. Flash-Next takes three of the five slots. The old local model keeps the local slot on log-triage, where the new one explicitly failed to displace it. Opus keeps the jobs where prose quality or diagnostic depth pays for itself. Model loyalty, pick a champion and route everything through it, means you ship worse results on some tools and pay frontier prices for no gain on others. The full per-tool numbers, method and caveats live in the report: Flash-Next vs Opus 4.7.

The other thing this pass confirmed is where the durable value sits. Models are passing through my estate at a rate of one serious candidate a month now. The thing that turns each arrival from a vibe into a decision is the harness: fixed corpus, pinned judge, archived judgments, a stability check before every comparison. That took real effort to build and it repays it every release. The frontier line and the local line will keep crossing each other tool by tool, and IMHO the estates that notice each crossing early will simply be the ones that measure. Mine took three days from model release to routing change. That loop is the asset. The models are just traffic.