Qwen 3.8-27B vs Claude Opus 4.7Copy link

This report carries the per-task numbers for the Qwen 3.8-27B versus Opus 4.7 pairing. The story these numbers tell, a shrinking gap between frontier and local on structured work, and the harness bugs caught before the scorecard could be trusted, is in the blog post The frontier gap, measured. The sibling pairing against Opus 4.6, which tests the head-to-head positioning in Qwen's own release benchmarks, is at Qwen vs Opus 4.6.

Engineering tasks	Real cases, both models	Qwen wins : ties : Opus wins	Opus API cost, 52 runs
5	52 x 2	2 : 1 : 2	$2.22

Headline metric per task for the ratio; Qwen ran free, on-prem, so its column of the cost line is $0.

VerdictsCopy link

Task	Verdict	Routing guidance
Code assessment	Adopt local	Perfect scores on both models on seeded bugs, zero false positives. Qwen does it at zero cost.
Structured extraction	Adopt local	Qwen beat Opus on schema validity (0.89 vs 0.78). Keep the validate-and-retry guardrail for both.
Test strategy	Adopt, routed	Qwen edges the F1 accuracy score (defined under the chart below) but runs 2x slower. Route by workload; raise its token and time budgets.
Commit / PR description	Opus first	Opus clearly better on faithfulness and tone. Qwen only for batch digests with a human reviewer.
Incident log triage	Opus first	Opus wins root-cause analysis (0.94 vs 0.81). Qwen is fine as a first-pass filter.

Quality by taskCopy link

Task	Metric	Qwen 3.8-27B	Opus 4.7
Test strategy	F1 vs reference scenarios	0.65	0.59
Code assessment	F1, seeded bugs	1.00	1.00
Commit description	Judge faithfulness (of 5)	4.08	4.75
Log triage	Root-cause match	0.81	0.94
Structured extraction	Schema validity	0.89	0.78

F1 blends precision (how much of what the model flagged was right) and recall (how much of what mattered it found) into one 0 to 1 score; only high when both are. Both arms were scored by the same fixed judge (Opus 4.7) on identical inputs, so numbers can be compared directly across this report and the Opus 4.6 sibling.

Speed by taskCopy link

Task	Qwen 3.8-27B (s/call)	Opus 4.7 (s/call)
Test strategy	85.5	43.7
Code assessment	20.6	5.7
Commit description	22.5	10.0
Log triage	30.5	22.0
Structured extraction	3.9	2.3

Mean seconds per call; shorter is better. Opus 4.7 runs 1.4 to 3.6 times faster per call. Qwen's throughput scales with concurrency on the local box, and its per-call cost is zero.

What we learned beyond the scoresCopy link

• Local wins where structure rules. Deterministic, schema-shaped tasks (code checks, extraction) are already local-grade. Judgement-heavy prose and causal reasoning still favour the frontier model.
• Pin exact model versions. A "latest" gateway alias silently ran our first pass on Opus 4.6. Every number here is from an exact-pinned 4.7 arm, re-judged for version consistency.
• Integration breaks before models do. Three client bugs (markdown-fenced JSON, a thinking-mode default, a deprecated parameter) made both models look broken before a single real quality comparison could run. All fixed and tested.
• Verify your harness, not just your model. An 8 KB pipe-buffer limit in the benchmark harness masqueraded as model failures. Skepticism about suspiciously round numbers paid for itself.

MethodCopy link

52 cases per model drawn from real repositories and incident history: seeded-bug diffs with clean controls, live operational logs, real commits, and deliberately unrepresentable extraction inputs. Qwen 3.8-27B NVFP4 served by vLLM on a DGX Spark with speculative decoding; Opus 4.7 via API at exact model pin, identical prompts. Objective metrics where they exist (precision and recall, schema validity); a single fixed judge (Opus 4.7) scored prose quality on both models in both this report and the Opus 4.6 sibling, so cross-page comparison holds. The self-preference caveat is recorded.

---

The sibling pairing against Opus 4.6 is at Qwen vs Opus 4.6. The narrative behind the numbers, and the harness bugs that had to be caught before the scorecard could be trusted, is in the blog post The frontier gap, measured.