Field report · gfx906 · speculative decoding

Two Retired GPUs, 55 Tokens per Second

What a week of block-diffusion drafting on a pair of Radeon Pro VIIs taught us about DFlash, hybrid models, and measuring honestly.

Sebastian Haas · August 2026 · 2× Radeon Pro VII (gfx906, 16 GB, ~150 € used each) · llama.cpp fork · all numbers greedy, temp 0

1The setup

The machine is deliberately unglamorous: two AMD Radeon Pro VII cards, the workstation twin of the MI50, gfx906, 16 GB HBM2 each, bought used for roughly the price of a mid-range CPU cooler habit, power-capped at 190 W, driving a llama.cpp-based fork with hand-tuned gfx906 kernels. Serving happens through a small Rust server speaking the OpenAI API; llama-server is used for A/B experiments.

The targets that matter here are hybrids: Qwen3.6-27B and Qwen3.8-27B interleave a few full-attention layers with many gated-DeltaNet (linear-attention) layers, and Qwen3.6-35B-A3B does the same as a MoE. Hybrids are exactly what you want on 16 GB cards, small KV cache, long contexts, and exactly what speculative decoding has trouble with, because recurrent state cannot be rewound when the verifier rejects drafted tokens. Our fork adds a bounded snapshot ring (“RS-ring”) that makes partial rollback exact within the draft window; that mechanism has its own write-up. This post is the practical story.

2The ladder

The 55.3 headline survived an adversarial re-measurement (n=8 exclusive runs, median 55.2, spread 0.2%, stable at 60 °C). More on why we needed an “adversarial re-measurement” in section 7.

3What DFlash is, in one paragraph

DFlash (z-lab, arXiv:2602.06036) replaces the usual autoregressive drafter with a tiny block-diffusion model: it fills a whole block of mask tokens in a single bidirectional forward pass, conditioned on hidden states pulled out of a few target layers. Version 1 (block 16) samples drafts from logits; version 2 (block 8, llama.cpp PR #27342) adds a trained confidence selector. The catch: every drafter is trained for one specific target model, and it borrows the target’s LM head. DFlash2 drafters exist only for Qwen3.8-27B and Muse-Glimmer-30B; v1 drafters cover much more (Qwen3.6 family, Qwen3.5 family, gpt-oss, Gemma-4, GLM, Kimi…). Community GGUFs: search Hugging Face for Alittlehammmer/*-GGUF-llama.cpp.

4Recipes that work on 16 GB cards

llama-server, Qwen3.8-27B + DFlash2

llama-server -m Qwen3.8-27B-Q4_0.gguf -c 8192 -ngl 99 -fa on --parallel 1 \
  --spec-type dflash -md Qwen3.8-27B-DFlash2-Q4_0.gguf \
  -devd ROCm1 -ngld 99 --draft-max 7 --draft-min 1 --draft-p-min 0.5

Three rules we learned the hard way:

• The drafter must sit on the GPU that holds the target’s last layers. It has no LM head of its own; put it elsewhere and graph scheduling aborts at load (pre-allocated tensor (output.weight) … cannot run the operation). Let the target split across both cards and pin the drafter to card 2.
• --parallel 1 on hybrid targets. The rollback ring costs ~150 MiB per snapshot per slot on a 27B; llama-server’s default of 4 slots tried to allocate 4.8 GiB and died.
• A 27B + drafter + ring never fits one 16 GB card. Two cards or nothing, but two cards it genuinely uses (as memory; layer-split adds no decode speed).

Single-GPU owners: take the 35B instead

The 55-mark does fall on one card, just not with the 27B. The 35B-A3B with its tiny drafter fits a single 16 GB GPU if you push one more expert block to the CPU (blocks 0–8; 0–7 dies on the compute buffer) at 8k context: 56.9 tok/s code / 59.1 prose on one card, 15.7 GiB used, second card fully idle. That’s ~14% off the two-card figure, and the freed card can run a second instance.

Quantize the drafter

llama-quantize --allow-requantize drafter-Q8_0.gguf drafter-Q4_0.gguf Q4_0

Acceptance is quantization-insensitive (65 vs 64% for us; the PR author measured BF16/Q8/Q4 at 4.92/5.08/5.03 accepted tokens per step). The drafter is bandwidth-bound, so Q4_0 is simply faster, it took our production number from 52.7 to 55.3. This holds for logit-sampling v1 drafters too.

v1 drafters (block 16)

Same flags, but --draft-max 15, block-16 drafters can propose up to 15 tokens, and the ring must be sized to match. The 35B-A3B drafter is a 386 MB downloadable and worth every byte on code.

5Scoreboard

Notice that every incumbent survives somewhere. MTP keeps prose and parallel load. Lookup keeps prose on the MoE (it feeds on repetitive n-grams and costs nothing). Block drafting owns code and reasoning. There is no single winner, which is the actual finding.

6The three walls we hit

Wall 1: drafters don’t transfer

Qwen3.8’s DFlash2 drafter attached to Qwen3.6-27B, same hidden size, same vocabulary, same layer count, runs fine mechanically and loses to baseline: acceptance drops from 64% to 42–49%. Hidden-state injection is a trained contract, not an interface. If z-lab didn’t train a drafter for your model, you don’t have a drafter. (Training your own is realistic only for v1, via DeepSeek’s DeepSpec repo; the DFlash2 training pipeline is proprietary.)

Wall 2: context depth

Our first port disabled the drafter’s sliding window to dodge a crash. Short contexts didn’t care; at 6.3k tokens deep, drafted decoding fell below baseline. Honoring the trained 2048-token window fixed both cost and acceptance (46 → 57%, the drafter sees the geometry it was trained with) and moved break-even to ~10–14k. At 14.9k, shortening drafts to --draft-max 3 reaches parity. Beyond that, switch speculation off. And check your drafter’s window: the 35B’s ships with 4096, which barely prunes at 6k, combined with CPU-offloaded MoE experts in every verify row, that one stays a net loss at depth no matter what.

Wall 3: parallel streams

Two concurrent streams through the block drafter aggregate to 41–51 tok/s; the boring MTP head aggregates 53–61. On CUDA-class cards the PR reports good multi-stream scaling; on gfx906, block drafting is a solo weapon. Batch serving keeps MTP.

7The measurement-integrity story

Midway through the campaign, our production numbers started degrading run over run, 45, then 27, then 20 tok/s. Classic thermal throttling, we noted, and added a cooling gate. Except it wasn’t. A second AI agent session, working the same backlog on the same machine, had started benchmarking the same GPUs at the same time. Two clients interleaving on a single-slot server halve each other’s numbers in a pattern that looks exactly like heat.

Untangling it produced the most transferable lessons of the week:

• Contention masquerades as thermals. If a number degrades monotonically, check for co-tenants before blaming temperature.
• The first response after model load is ~10% slow. Reproducibly (50.3 vs 55.x for us). Warm up, then discard the warm-up.
• Advisory locks work, with flock -o. Without -o, a daemonized server started under the lock inherits the file descriptor and holds the lock forever. We blocked ourselves for ten minutes before finding that one.
• Kill by remembered PID, never by pkill pattern. Our two sessions shot down each other’s servers for half an hour, producing one SIGKILL mystery and two “unexplained” crashes that were never bugs at all.
• Settle disputes with exclusive arbitration runs, cooling-gated, n≥8, lock held, co-tenant confirmed idle. Every disputed number either reproduced exactly or dissolved into one of the artifacts above.

If you run autonomous agents against real hardware, they need the same hygiene as a shared cluster. Nothing about “it’s just my workstation” exempts you.

8When to use what

One subtlety worth stealing: reasoning traces are their own content class. Same acceptance rate as prose, 1.5× the throughput, because “thinking” output is locally predictable for long stretches and drafts run to full depth. We mis-read one of our own benchmarks for half a day because the serving stack in one arm had reasoning mode on and the other off. Label your content class, or your numbers will lie to you politely.

9Reproducing this

Models: Qwen3.8-27B / Qwen3.6-27B / Qwen3.6-35B-A3B GGUFs plus their z-lab drafters (Hugging Face; v1 community GGUFs by Alittlehammmer, DFlash2 convertible from safetensors). Code: llama.cpp PR #27342 for DFlash2; our numbers additionally sit on a gfx906-tuned fork with the recurrent-state rollback ring, so treat the absolute figures as this-hardware-with-this-fork. The ratios, drafter ranking, content-class split, depth break-even, quantization insensitivity, are the parts we’d expect to travel. Full benchmark logs live in the project’s BENCHMARKS.md (campaigns CR–CW).

Written from one week of measurements on one machine, with substantial AI assistance operating the workstation under the author’s direction. Companion technical report: “Speculation Needs Rollback” (RS-ring mechanism and the full measurement matrix). Corrections welcome, especially numbers from other MI50/Pro VII owners.