Three models, one draft head
Two Qwen3.8-27B variants (original and abliterated) and one Qwen3.6-27B run on the same 150-watt MI50, in the same format, on the same stack. At the kernel level, their model bodies are indistinguishable, yet their throughput differs by 10%. This study isolates the difference to a single component: the supplied multi-token prediction head and whether it ever learned to keep predicting from its own predictions.
between raw models (llama-bench)
from the better head
of abliteration
second draft stage
Abstract
Speculative decoding with a model’s own multi-token prediction heads (MTP) is often treated as an implementation problem. Using three architecturally identical 27-billion-parameter models, we show that the achievable gain is instead a model property. All three variants were measured in the same quantization format (Q4_0), on the same hardware, and with the same serving stack: perplexity on WikiText-2 (32 chunks), first- and second-stage draft acceptance, and end-to-end throughput at speculation depths 1 and 2. The raw models are indistinguishable at the kernel level (tg64 = 32.3–32.5 t/s, Δ < 0.7%), yet their serving rates differ by 10% (45.8 versus 50.3 t/s). The cause is the acceptance rate of the chained second draft stage: the Qwen3.8 head loses 19 percentage points relative to the first stage (92.9 → 74.2%), while the Qwen3.6 head loses only 6–9 (96.1 → 87–90%). A separately published replacement head reproduces exactly the weaker values, ruling out quantization and fine-tuning as causes. As a secondary result, we quantify the cost of abliteration (+1.2% perplexity in the symmetric pair) and document two measurement traps that systematically distort acceptance rates and perplexity values.
- The model body is not the deciding factor: both 27B models have the same raw speed (32.4 t/s); the serving difference comes entirely from the draft head.
- The second stage is what matters. Only a head that can keep predicting from its own output earns speculation depth 2. The 3.6 head can; the 3.8 head cannot.
- Abliteration costs 1.2% perplexity and leaves the draft head untouched.
- Two measurement traps: acceptance rates also reflect the generated text; perplexity is only comparable with identical chunk counts (16 versus 32 chunks: a 2-point difference).
Three candidates, one suspicion
The Qwen3-27B hybrid combines 48 Gated DeltaNet layers with 16 conventional attention layers and includes an additional NextN block: a trained head that predicts the next token from an intermediate state already computed. In speculative operation, this head guesses, and the large model verifies the proposal in the same pass. A correct guess yields two tokens for the price of one.
This mechanism had been running on our card for weeks. But adding a second, chained draft stage revealed a pattern that did not fit the hardware: it contributed 2.6% on one model and 12.6% on another, with identical architecture, format, and code. This study explains the difference.
| Candidate | Source | File (Q4_0) | Distinguishing feature |
|---|---|---|---|
| Qwen3.8-27B base | unsloth | 14.95 GiB | Reference, unchanged |
| Qwen3.8-27B Uncensored | mradermacher, imatrix | 14.60 GiB | Abliterated |
| Qwen3.6-27B | unsloth, MTP variant | 14.95 GiB | Earlier generation, native MTP head |
How we measured, and what can go wrong
All runs used an AMD Radeon Pro VII (gfx906, 16 GiB, hard 150-watt limit), ROCm 10.1, a custom llama.cpp fork with gfx906-optimized Q4_0 kernels, and a Rust serving stack with an MTP verification loop. A cooling gate preceded every run (edge temperature ≤ 44 °C); each cell used three repetitions and the median. Thermal outliers are reported, not silently discarded.
Three metrics: perplexity (WikiText-2, 32 chunks, -ngl 99 -fa 1) as a quality anchor; acceptance rates α₁ and α₂ through a probe that generates a chained second draft from the first and then removes its probe row from the cache; and end-to-end throughput in serving at temperature 0.
Two traps, both encountered and corrected during this study:
(1) Acceptance rates also measure the text. The same model file showed α₁ = 88.6% before and 90.7% after an output requantization step. Requantization slightly changes the logits, which changes the generated text, and more predictable text has higher acceptance. An initially recorded “head advantage” for the abliterated model disappeared in exactly this way. Only runs producing the same text are comparable.
(2) Perplexity depends on chunk count. The same file scores 7.76 over 16 chunks and 5.84 over 32; WikiText contains sections of different difficulty. Comparing across different chunk counts is meaningless. We too initially “measured” a 3% quality loss this way, which shrank to 0.5% with a properly matched comparison.
The measurement matrix
All cells use the same format, the same card, and the same chunk count. Serving d2 denotes throughput with two-stage speculation on a code prompt; B2 an output embedding additionally requantized to Q4_0, pushing the file below the 16-GiB card’s residency threshold.
| Variant | Size | PPL₃₂ | Serving d2 | α₁ | α₂ |
|---|---|---|---|---|---|
| 3.8 Uncensored Q4_K_M | 15.6 GiB | 5.5997 | 27.0 (offload) | ||
| 3.6 B2 | 14.3 GiB | 5.6553 | 50.3 | 96.1% | 87–90% |
| 3.8 base Q4_0 | 14.95 GiB | 5.6942 | 38.0 | 88.1% | 76.1% |
| 3.8 base B2 | 14.3 GiB | 5.7722 | 46.0 | 93.2% | 73.5% |
| 3.8 Uncensored i1-Q4_0 | 14.6 GiB | 5.8041 | 42.0 | 88.6% | 73.6% |
| 3.8 Uncensored B2 | 14.3 GiB | 5.8404 | 45.8 | 90.7% | 72.5% |
Why the second stage decides
On the first draft, the head receives ideal input: the trunk’s actual intermediate state and the actual last token. All three heads can handle this (α₁ = 88–96%). On the second, chained draft, the situation fundamentally changes. It now receives its own intermediate state and its own guessed token as input. It runs on its own output, and small errors from the first stage are already embedded in the second stage’s input.
The economics are a simple balance. An additional verification row costs a measured 25% of step time on this hybrid; the linear DeltaNet scan across 48 layers scales with row count. In expectation, it adds α₂ tokens:
E[tokens/step] = 1 + α₁ + α₁·α₂ · Cost = t₂ · 1.25
With α₁ = 0.93 and α₂ = 0.74 (Qwen3.8), a +28% ceiling faces +25% cost: effectively a wash, confirmed by the measured +2.6%. With α₂ = 0.90 (Qwen3.6), the same calculation shifts to +34% against +25%, and measurement delivers +12.6%. The only changed parameter is second-stage acceptance.
Cross-check: could quantization be responsible after all? The suspicion was plausible because Uncensored uses an imatrix recipe while base does not. The probe across three Qwen3.8 variants, imatrix-Q4_0, IQ4_NL, and Q3_K_M, yields α₂ = 72.7 / 73.0 / 73.6%. A span of 0.9 points across three different quantization methods. When the manufacturer also published the head as a separate 1.37-GB file, a transplant was therefore pointless: it is the same head, and it is the limit.
What does abliteration cost?
The second comparison in this study concerns fine-tuning. Earlier measurements with different quantization recipes had shown “no measurable difference,” a result refined by the symmetric comparison. With identical format and requantization state, abliteration costs:
| Pair | base | Uncensored | Increase |
|---|---|---|---|
| Q4_0 original | 5.6942 | 5.8041 | +1.9% |
| B2 (symmetric) | 5.7722 | 5.8404 | +1.2% |
A 1.2% increase is smaller than the cost of a quantization step (Q4_0 versus K-quant of the same model: 4.3%) and well below variation between model generations. More importantly in practice: the draft head remains intact. α₁ and α₂ are statistically indistinguishable between base and abliterated (88.1/76.1 versus 88.6/73.6). If you need uncensored behavior, you pay in perplexity, not speculative gain.
What this study does not show
Perplexity is not a measure of usefulness. WikiText-2 measures prediction of encyclopedic prose. Models tuned more heavily for instructions and code regularly show higher scores there while performing better on tasks. Qwen3.6’s 3% advantage is therefore not evidence of greater everyday usefulness. Qwen3.8 is the newer generation, and a task-based comparison remains outstanding.
The error bars overlap. At 32 chunks, uncertainty is ±0.15; gaps between neighboring matrix rows are smaller. What is robust is the consistent direction across multiple chunk counts and pairings, not the third decimal place of a single cell.
One hardware context, one format. All statements apply to Q4_0 kernels on gfx906 under a 150-watt limit. On hardware where a verification row is cheaper than here (conventional transformers without a linear scan, or cards with more memory bandwidth), second-stage economics shift in favor of speculation. Even a weaker head may then pay off.
Speculative gain is trained acceptance
The common view that speculative decoding is an implementation question falls short. With identical model bodies, kernels, and serving stacks, the quality of the supplied draft head alone determined a 10% throughput difference, specifically through a property no standard benchmark reports: how well the head can keep predicting from its own prediction.
This leads to an uncomfortable practical recommendation: if you plan to use speculative decoding, measure α₂ before choosing the model. The probe takes two minutes and one prompt; a wrong choice costs half the possible gain. In our case, this exact number overturned a rigorously calculated “do not build” verdict from eight days earlier. Not because the calculation was wrong, but because its underlying cost basis had shifted.
Operating recommendation for this card. Qwen3.6-27B with the B2 recipe as the default (5.6553 at 50.3 t/s, the best combination of quality and speed). Qwen3.8-Uncensored-B2 when uncensored behavior is needed: a 1.2% perplexity surcharge, 45.8 t/s, fully resident. Qwen3.8-base-Q4_K_M when quality takes priority over speed and partial offload is acceptable.
All raw data in BENCHMARKS.md (campaigns AT.4–AT.8, AU), with the timeline in WORKLOG.md. Measurement tools in the fork: MTP_D2_PROBE=1 (second-stage acceptance), MTP_TAIL_CMP=1 (draft comparison of two compute paths). 34 measurement cells, three repetitions per cell, cooling gate between all runs.