Qwen 3.8-27B vs Claude Opus 4.6Copy link

This report carries the per-task numbers for the Qwen 3.8-27B versus Opus 4.6 pairing, the head-to-head Qwen's own release benchmarks put on the table. The vendor's numbers, from the model card: Qwen ahead on SWE-bench Pro (61.7 vs 53.4), behind on Terminal Bench 2.1 (73.0 vs 78.2) and GPQA Diamond (89.2 vs 91.3). The story these numbers tell, how far the model card's positioning survives contact with real work, and the harness bugs caught before the scorecard could be trusted, is in the blog post The frontier gap, measured. The sibling pairing against the current frontier, Opus 4.7, is at Qwen vs Opus 4.7.

Engineering tasks	Real cases, both models	Qwen wins : ties : Opus wins	Opus API cost, 52 runs
5	52 x 2	1 : 1 : 3	$1.04

Headline metric per task for the ratio; Qwen ran free, on-prem, so its column of the cost line is $0.

VerdictsCopy link

Task	Verdict	Routing guidance
Code assessment	Adopt local	Perfect scores on both models on seeded bugs; a dead heat at the ceiling. The vendor positioning holds, and the local model does it at zero cost.
Test strategy	Adopt, routed	Qwen wins the F1 accuracy score outright (0.65 vs 0.59; defined under the chart below) but runs at roughly double the per-call time. Route by workload.
Structured extraction	Adopt local	Split result: Opus 4.6 wins schema discipline (1.00 vs 0.89), Qwen wins content accuracy (exact match 0.56 vs 0.44). Keep the validate-and-retry guardrail for both.
Commit / PR description	Opus first	Opus 4.6 ahead on faithfulness, completeness and tone (4.75 vs 4.08 headline); the vendor positioning does not hold here. Qwen only for batch digests with a human reviewer.
Incident log triage	Opus first	Opus 4.6 wins root-cause (0.88 vs 0.81) and next-step quality; Qwen surprisingly wins anomaly plausibility. Qwen is fine as a first-pass filter.

Quality by taskCopy link

Task	Metric	Qwen 3.8-27B	Opus 4.6
Test strategy	F1 vs reference scenarios	0.65	0.59
Code assessment	F1, seeded bugs	1.00	1.00
Commit description	Judge faithfulness (of 5)	4.08	4.75
Log triage	Root-cause match	0.81	0.88
Structured extraction	Schema validity	0.89	1.00

F1 blends precision (how much of what the model flagged was right) and recall (how much of what mattered it found) into one 0 to 1 score; only high when both are. Both arms were scored by the same fixed judge (Opus 4.7) used in the Opus 4.7 sibling report, so the two pages compare directly.

Speed by taskCopy link

Task	Qwen 3.8-27B (s/call)	Opus 4.6 (s/call)
Test strategy	85.5	46.3
Code assessment	20.6	7.7
Commit description	22.5	11.5
Log triage	30.5	26.4
Structured extraction	3.9	2.9

Mean seconds per call; shorter is better. Opus 4.6 runs 1.2 to 2.7 times faster per call. Qwen's throughput scales with concurrency on the local box, and its per-call cost is zero.

What the pairing tells usCopy link

• The vendor positioning is task-shaped, not model-shaped. It holds for structured, ground-truth work and breaks on judgement-heavy prose. Route by task, not by slogan.
• Conservatism pays on strategy. Qwen's higher-precision style beats both frontier versions on test-strategy F1. Sometimes proposing less, more accurately, is the winning move.
• Newer is not uniformly better. Opus 4.6 held perfect schema discipline where 4.7 slipped to 0.78. Version upgrades shift behaviour; re-verify what you rely on.
• The frontier premium is narrow but real. Where Opus wins, it wins on the axes humans feel first: faithfulness of prose and quality of diagnosis. That is what the API fee buys.

MethodCopy link

Identical to the Opus 4.7 evaluation: 52 cases per model from real repositories and incident history, Qwen 3.8-27B NVFP4 served by vLLM on a DGX Spark with speculative decoding, Opus 4.6 via API at exact model pin, identical prompts. Objective metrics where they exist (precision and recall, schema validity); a single fixed judge (Opus 4.7) scored prose quality for both models in both reports, so cross-page comparison holds. The self-preference caveat is recorded.

One asymmetry noted for completeness: the Opus 4.6 outputs were generated at a fixed low temperature, the Opus 4.7 outputs at the provider default. Both sit in the ranges the respective providers use for coding work, and the direction of effect (lower temperature narrows variance rather than shifting the mean of judged quality) does not change the verdicts, but it is worth naming.

---

The sibling pairing against the current frontier is at Qwen vs Opus 4.7. The narrative behind the numbers, and the harness bugs that had to be caught before the scorecard could be trusted, is in the blog post The frontier gap, measured.