The frontier gap, measured.Copy link

The gap between the frontier models and the open-weight models you can run on your own hardware is closing faster than I'd realised. That's an easy claim to type and a hard one to defend, so here's a concrete pin. Qwen's own model card puts Qwen3.8-27B, a model that fits on a desktop machine, head-to-head against Claude Opus 4.6, and the vendor's tables trade wins: ahead on one benchmark, behind on two. Vendor tables are marketing until someone checks them on work they actually care about, so I checked.

I had the means and the motive. I've built five small tools, each wrapping a language model, that run unglamorous jobs for me all day (code review, log triage, that sort of thing), and each one needs a decision about which model it calls. I also have an NVIDIA DGX Spark sitting on my desk. So I ran the card's positioning against my own tasks, on my own hardware, against both Opus 4.6 and 4.7.

Local takes three of the five verdicts, the frontier keeps two, and neither side is embarrassed. A model running at zero marginal cost on my desk is holding its own against the top of the price list on real engineering work. The scorecard is below, and after it, the checking I did before I believed it, because my own benchmark tried to lie to me on the way.

What I was testingCopy link

Five tasks, each a single-purpose command-line tool around one model call. code-assess reads a code diff and flags bugs. extract-structurise turns freeform text into strict JSON, or refuses when the text doesn't fit the schema. change-describe writes the human-readable description of a code change. test-strategy reads a module and proposes what to test. log-triage reads a window of production logs and says what went wrong and what to check next.

The corpus is real wherever it can be: actual commits, production logs and code from my own working repos, plus hand-seeded bug diffs where the ground truth has to be planted to be knowable. 104 runs across two arms, byte-identical prompts, only the model behind the endpoint differs. Arm A is Qwen3.8-27B on the Spark. Arm B is Claude Opus, pinned to the exact model ID at every invocation and every judge call. Where a metric has objective ground truth (did it find the seeded bug, does the JSON validate) it's scored directly. Where it doesn't (is this commit description faithful, is that triage useful) a judge model scores both arms against a fixed rubric with the arm labels stripped out.

The honest three-way verdictCopy link

Same corpus for every arm, one judge (Opus 4.7 grading all three arms), exact model pins everywhere. This is where it lands.

code-assess is a dead heat. All three arms score a perfect F1 (the balance of precision and recall) on the seeded-bug corpus, with zero false positives on the clean-diff control. When quality ties, marginal cost decides, and the local box is already paid for. Local.

extract-structurise goes to Qwen against 4.7 on schema validity, 0.89 vs 0.78. The wrinkle is that 4.6 scored a perfect 1.00 on the same corpus, so frontier generations don't move in one direction. Route it local with a schema-validate-and-retry guardrail and the residual risk on either model is covered.

test-strategy gives Qwen a small quality edge, F1 0.65 against 0.59 for either Opus, at roughly double the latency. Workload-routed: local when the caller can wait, frontier when it can't.

change-describe is the frontier's tool outright. Opus wins faithfulness, completeness and tone, and the gaps aren't small. Writing prose a human will read is still frontier work. It stays.

log-triage also goes frontier where it counts. Root-cause match runs 0.94 for Opus 4.7, 0.88 for 4.6, 0.81 for Qwen, and for incident work I want the model most likely to name the actual cause. Qwen is fine for the background sweeps where a human re-triages the digest anyway.

So the card ends three to two in local's favour. The full per-tool numbers, method and caveats live in two reports: Qwen vs Opus 4.6 answers the model card's head-to-head on my own tasks, and Qwen vs Opus 4.7 is the same corpus against the current frontier.

Before you trust the scorecardCopy link

An even scorecard from a home-built benchmark deserves suspicion, because a benchmark has a way of flattering the model its author wants to win. Mine had a go at exactly that. It lied to me more than once on the way to that table, every lie produced plausible-looking numbers, and the two big ones would both have flattered the local model. Here's what got caught, and what to watch for in your own harness.

The first lie lived in the harness. Four Opus cases on test-strategy landed in the metrics as failures. They weren't. On macOS, spawnSync with piped stdout truncates at around 8188 characters, and the larger Opus responses came back half-clipped, so the judge marked them incomplete. The tell was the cluster: four failures, all near the same byte boundary. Real failures don't queue up at a round number. The fix was streaming to a real file descriptor instead of buffering a pipe. The lesson is older than language models: when the numbers surprise you, suspect the harness before the model.

The second came from the vendor's side. Every tool sent temperature: 0.1 on every call, because that's what the shipped defaults did. Opus 4.7 rejects it with an HTTP 400: temperature is deprecated for this model. Opus 4.6 accepts it happily. Resend without the parameter and it works. Any harness that assumes parameters behave the same across model versions falls over on the next release, and it falls over on the day the vendor moves, not the day you change something.

The third was the price list. Opus 4.6 costs $3 per million input tokens and $15 per million output. Opus 4.7 is $5 and $25. A cost-per-call baseline computed six weeks ago is already wrong, and cost was half the reason for the benchmark. If a routing decision cites a dollar figure, date the figure.

And two smaller ones from setup, one for each side. Qwen ships with thinking mode on by default, and on tool-shaped tasks it will spend the entire completion budget reasoning and hand you an empty answer; left alone, that one sinks the local arm. I turned thinking off for structured work. The proper fix is probably a capped thinking budget rather than the off switch; I took the blunt route because the benchmark needed deterministic structured output, and I haven't tested where a sensible cap lands. Opus, asked for a JSON object via response_format, routinely wraps the JSON in a triple-backtick fence anyway; strip the fence in the client rather than failing the parse. The format parameter is a hint, not a contract.

A benchmark is a piece of software, and it has bugs like any other piece of software, except its bugs come out dressed as findings. Nothing about the truncation lie looked broken; it produced a clean table. The habits that caught it were boring ones: pin exact model versions everywhere including the judge, treat suspicious clusters as harness defects until proven otherwise, and date your cost figures. With those fixed, the scorecard above is as honest as I know how to make it.

The box on my deskCopy link

The local arm only made this a contest because of a tuning pass that roughly doubled its throughput, so here's the working configuration in full.

The server is vLLM, image pinned to vllm/vllm-openai:v0.24.0-ubuntu2404, the DGX Spark recipe build. The checkpoint is unsloth/Qwen3.8-27B-NVFP4, a community 4-bit requant. One trap on the way in: the nvidia-branded nvidia/Qwen3.8-27B-NVFP4 doesn't exist on Hugging Face (it 404s even with an authenticated token), so the community requant is the one to pull. It also ships the extra draft-head weights that make speculative decoding possible, which matters below.

command:
  - unsloth/Qwen3.8-27B-NVFP4
  - --served-model-name
  - qwen38-27b-nvfp4
  - --download-dir
  - /root/.cache/huggingface
  - --max-model-len
  - "131072"
  - --gpu-memory-utilization
  - "0.60"
  - --max-num-seqs
  - "8"
  - --max-num-batched-tokens
  - "8192"
  - --enable-chunked-prefill
  - --async-scheduling
  - --enable-prefix-caching
  - --speculative-config
  - '{"method": "qwen3_5_mtp", "num_speculative_tokens": 3}'
  - --load-format
  - fastsafetensors
  - --enable-auto-tool-choice
  - --tool-call-parser
  - hermes
  - --reasoning-parser
  - qwen3
  - --trust-remote-code
  - --limit-mm-per-prompt
  - '{"image": 0, "video": 0}'

The flag doing the real work is --speculative-config. This is multi-token prediction, MTP: the checkpoint carries a small draft head that guesses the next few tokens, and the main model verifies the guesses in one step instead of generating one token at a time. vLLM logs a deprecation warning that qwen3_5_mtp is remapped to the generic mtp method. Harmless, same code path.

The before and after. Stock configuration decoded at 11.6 tokens per second single-stream and 82.8 aggregate across eight concurrent streams. With MTP at three speculative tokens plus one more change, single-stream lands between 20 and 28 tokens per second depending on content (+75% on prose, +144% on code, +124% on maths) and 129 aggregate at eight streams (+56%). Time to first token roughly doubles, from 138ms to 273ms, which is the expected price of speculative decoding: the draft-and-verify step runs before the first token can come out.

The other change was dropping --kv-cache-dtype fp8, so the KV cache (the model's working memory of the context so far) stays at its native precision. That gains 4 to 12% single-stream. The cost is headroom, down from about 45 GiB and 1.37M cacheable tokens to about 22 GiB and 685K, still ample for this workload. I also tested --async-scheduling off and lost 3 to 8% under concurrency, so it stays on.

For calibration against the community: the popular wrapper project for this checkpoint self-reports about 21.4 tokens per second single-stream, the official vLLM recipes walkthrough for DGX Spark NVFP4 about 24.5, and an SGLang NVFP4 setup about 34. This config sits inside that 21 to 34 band on code and maths.

Two warnings that will save you an evening. min_p and logit_bias don't work with speculative decoding, and nothing tells you at request time. And --tool-call-parser hermes does not parse this model's XML-style <function=X> tool-call output; the model emits the call correctly and the parsed tool-call array comes back empty. If you need native tool-calling, check the parser inventory before trusting it.

Where this landsCopy link

The model card's positioning holds where the work is structured (finding bugs, extracting fields, listing test targets) and doesn't where the work is prose and diagnosis. That's a more useful answer than a winner, because it routes. The high-volume calls go to the box on the desk at zero marginal cost, and the calls where quality is the point stay frontier.

Be careful what you carry out of here, though. These numbers don't say open weights have caught up, and I'm not claiming they do. Where the frontier wins on my corpus, it wins on the axes humans feel first, faithfulness of prose and quality of diagnosis, and that premium is what the API fee buys. What the numbers do say is narrower and more interesting: on structured engineering work with checkable ground truth, a 4-bit 27B on a desktop machine is already a peer, close enough that the routing question is live on every tool. I didn't expect the table to come out that even when I set this up.

These verdicts have a shelf life, too. Prices moved mid-benchmark, a parameter got deprecated mid-benchmark, and frontier generations don't even move in one direction: 4.6 held perfect schema discipline where 4.7 slipped. I'll rerun this when the next Qwen release lands. But right now the gap between the frontier and the box on my desk is two tools out of five, prose and diagnosis, and that's a much shorter distance than I would have guessed before I measured it.