Qwen3.8-Flash-Next vs Claude Opus 4.7Copy link

This report carries the per-task numbers for Qwen3.8-Flash-Next versus Claude Opus 4.7, with Qwen3.8-27B held alongside as the incumbent local baseline. The narrative behind the numbers, and the model release that provoked it, is in the blog post The local model won one of my five tools.. The sibling pairing of the 27B against Opus 4.7 on the same corpus and same judge is at Qwen vs Opus 4.7.

Engineering tasks	Real cases, three models	Flash-Next wins : ties : Opus wins	Opus API cost, 52 runs
5	52 x 3	1 : 2 : 2	$2.23

Headline verdict per task for the ratio; both local arms (27B and Flash-Next) ran free on-prem, so their columns of the cost line are $0. Flash-Next replaces 27B as the local default on three of the five tools; 27B keeps the local slot on log-triage, where Flash-Next explicitly regresses.

VerdictsCopy link

Task	Verdict	Routing guidance
Test strategy	Adopt Flash-Next	F1 0.791 leads 27B by 0.117 and Opus 4.7 by 0.166 on the same corpus and judge. Precision-driven win (0.749 vs 0.552 vs 0.459). Runs 37% faster than 27B on this tool.
Code assessment	Adopt Flash-Next (local default)	Perfect F1 on all three arms, zero false positives on the control. Flash-Next is 1.9x faster than 27B on latency; Opus 4.7 is still fastest at 5.7s.
Structured extraction	Adopt Flash-Next (local default)	Parity with 27B on every quality axis (schema validity 0.89, exact match 0.56, field recall 0.84). Keep the validate-and-retry guardrail; Opus 4.7 still trails on schema validity.
Commit / PR description	Opus first	Opus wins faithfulness, completeness and tone by wide margins (4.83 / 4.58 / 3.83 vs 4.00 / 3.33 / 2.83). Local arms interchangeable on this task; Flash-Next replaces 27B for batch fallback.
Incident log triage	Opus first, 27B for batch	Flash-Next regresses on every judged axis vs 27B (root-cause 0.63 vs 0.81 vs 0.94). Do not route to Flash-Next. Regression concentrates on the longest log windows.

Quality by taskCopy link

Task	Metric	Qwen3.8-27B	Qwen3.8-Flash-Next	Opus 4.7
Test strategy	F1 vs reference scenarios	0.674	0.791	0.625
Code assessment	F1, seeded bugs	1.00	1.00	1.00
Commit description	Judge faithfulness (of 5)	4.08	4.00	4.83
Log triage	Root-cause match	0.81	0.63	0.94
Structured extraction	Schema validity	0.89	0.89	0.78

F1 blends precision (how much of what the model flagged was right) and recall (how much of what mattered it found) into one 0 to 1 score; only high when both are. Every judged score in this report was graded by the same fixed judge (Opus 4.7 at exact pin) so numbers can be compared directly across this report and the 27B sibling. The change-describe row is from the new three-way blind judge; the paired A-vs-B judgments used in the sibling report agree with these numbers to within 0.25 on every axis.

Speed by taskCopy link

Task	Qwen3.8-27B (s/call)	Qwen3.8-Flash-Next (s/call)	Opus 4.7 (s/call)
Test strategy	85.5	53.6	43.7
Code assessment	20.6	10.7	5.7
Commit description	22.5	17.0	10.0
Log triage	30.5	38.5	22.0
Structured extraction	3.9	5.1	2.3

Mean seconds per call; shorter is better. Flash-Next is faster than 27B on three of the five tools and matches Opus 4.7's shape but not its throughput. Log triage is the one tool where Flash-Next loses on wall-time as well as quality: 26% slower than 27B on the largest-prompt task in the benchmark.

What we learned beyond the scoresCopy link

• Route the tool, not the estate. A 6B-active-parameter model beat a frontier model on one production tool and lost to everything on another, in the same estate, in the same week. Neither "local has caught up" nor "local is a toy" survives the per-tool numbers.
• Rejudge before comparing. Before scoring the new arm against anything, the pinned judge was re-run over the two previous arms. Deltas of 0.02 to 0.04 F1, inside the noise band the earlier passes documented. Skipping this step means comparing a real change to judge drift.
• Precision is the tool-planning axis. Flash-Next's test-strategy win comes from proposing fewer low-value cases, not from covering more. The frontier's near-perfect recall is bought by proposing nearly everything, and the judge marks it down for exactly that.
• The failure mode on log triage is confident-and-wrong. Schema validity held at 100%; envelopes stayed well-formed. The diagnosis inside the envelopes weakened, concentrated on the longest, multi-hour log windows. A model with 6B active parameters trying to hold a long operational narrative together is the shape of that regression.
• Serving-side surprises are not in the model card. The recipe crash-looped six times at 93% loaded because the memory-map patch wanted a local snapshot path, not a model id. Box memory accounting overshoots its configured budget by roughly 30GB. Plan against measured numbers.

MethodCopy link

52 cases per model, unchanged from the sibling report: 10 seeded-bug diffs with a clean control, 12 hand-crafted schema pairs (9 representable, 3 unrepresentable), 12 real operator-repo commits, 10 module and test-file pairs, and 8 windows of internal production logs. Flash-Next served by vLLM on a DGX Spark (GB10, 128GB unified memory) from checkpoint RadixArk/Qwen3.8-Flash-Next-NVFP4 at snapshot 7b719225242aacd3dbd3f9407468c2ee9a9d2594, routed through the internal LiteLLM gateway as spark/qwen38-flash-next-nvfp4. Serving-side concurrency ceiling 2; harness ran at the same. enable_thinking = false injected via chat_template_kwargs on every arm-C call, matching the 27B baseline. Opus 4.7 via API at exact model pin; the temperature parameter stripped at the vendor edge per the 4.7 shim. Judge held constant across all three arms and both reports: Opus 4.7 at exact pin, arm labels blinded, one fixed rubric per tool. change-describe used a new three-way blind judgment with slot assignments shuffled per case; paired judgments are preserved on disk alongside. Rejudge stability check before scoring: arm A and arm B rejudged on the two judge-scored tools, deltas 0.02 to 0.04 F1, inside the previously documented noise band. Local arms zero marginal cost on already-sunk hardware.

Two integration lessonsCopy link

Two harness issues landed in this pass; both caught before any metric baked.

The gateway route spark/* sits outside the local/* regex in the shared client, so it did not inherit the enable_thinking = false auto-injection. A first smoke test consumed 33 reasoning tokens on a "say ok" prompt; the fix was to force the kwarg on the caller side. Lesson: when adding a new local route outside the existing prefix, either extend the client regex or push the kwarg on every request.

The judge call still shipped temperature: 0.0, which Opus 4.7 rejects at the vendor edge. The SOP note about omitting temperature was already written; the code had not caught up. Twelve HTTP 400s in a row before the fix. Lesson: SOP notes about already-fixed harness bugs are worth checking against the actual current source before a run.

---

The sibling pairing of the 27B against Opus 4.7 is at Qwen vs Opus 4.7. The narrative behind these numbers is in the blog post The local model won one of my five tools..