Running Qwen3.8 on a deprecated GPU
A 2018 datacenter card with 16 GB of HBM2, bought used for about a hundred euros, runs a dense 27-billion-parameter hybrid entirely in VRAM at 21 tokens per second. Two open GitHub issues report the same models crashing on the same hardware. Neither the card nor our fork explains the difference, 35 MB of missing kernels does. This is the full field guide, including everything that still breaks.
fully resident on 16 GB
(Q3_K_M with a q8_0 KV cache)
head, 100% acceptance on code
hardware cannot run these models
Two bug reports say this is impossible
This investigation started with an offhand question from the machine's owner, after weeks of kernel work on the card: “Could you check whether anyone has actually got this running? I always assumed it wasn't possible.”
We had been running a dense 27B model on it for days. Nothing about it felt exotic. The search found something else.
Issue #19880, five models, one error
Filed February 2026 against llama.cpp by an owner of three MI50 32 GB cards. Still open, still unconfirmed, no replies. Qwen3.5-27B, Qwen3.5-35B-A3B, Qwen3.5-122B-A10B and both Qwen3-Coder-Next variants all abort with the same message:
rocBLAS error from hip error code: 'hipErrorInvalidDeviceFunction':98Vulkan works. ROCm does not.
Issue #19975, segfault in warmup
Two MI50, ROCm 7.2, build b8179. Segmentation fault during the warmup run, before the first token is produced. Marked as a duplicate.
Four of the models listed across those two reports are in our benchmark table with numbers rather than stack traces. So the interesting question is not whether the card can do it, but what exactly is different about our machine.
The difference is a library, not the card
hipErrorInvalidDeviceFunction is more specific than it looks. It means
a program tried to launch a GPU kernel that does not exist for this
architecture. Not “computed wrong”, not “too slow”, absent.
The caller is rocBLAS, AMD's matrix-multiplication library. It ships pre-compiled kernels per architecture, generated by a tool called Tensile. If the gfx906 set is missing, every call routed through it dies, and the newer Qwen hybrids route more work through it than older dense models did.
We did not learn this from documentation. We found it in a profile taken for an
entirely different reason. While mapping 144 matrix shapes to see which tiling
rocBLAS picks internally, the kernel names showed up in the rocprofv3
trace:
Cijk_Alik_Bljk_HB_MT128x128x16_SN_...Tensile kernels for gfx906, built and present. We had been benchmarking them for weeks without realising that their mere existence is the thing that fails elsewhere.
The one command to run before anything else
ls /opt/rocm/lib/rocblas/library/
On a working setup a gfx906 directory appears. Ours holds
210 files, 35 MB of Tensile kernels, and it is the only
architecture directory in the install, because TheRock builds per target. If that
directory is missing or empty you will hit issue #19880's error regardless of which
llama.cpp build you use, and no amount of kernel tuning will help.
Where ours comes from: AMD's TheRock nightlies, release layout
10.1, HIP runtime 7.16.26314. Distribution packages tend
to economise on build time for deprecated architectures, and gfx906 has been
formally deprecated for years. You find out what got dropped only when a model
happens to need that exact path.
An independent witness
This is not only our inference from a profile. An owner running two MI50 32 GB cards documents the same mechanism from the other side, in a different piece of software: their working stack is “rocBLAS 5.7.3 with gfx906 tensor files”, and they note that “Ollama v0.13.0 omits gfx906 files; v0.12.3 includes them”. Same library version as ours, same failure mode, observed independently.
One honest qualifier. The stack trace in issue #19880 passes
through ggml_cuda_graph_evaluate_and_capture(), the crash surfaces
during HIP graph capture. hipErrorInvalidDeviceFunction remains the
signature of a missing kernel, and the evidence above supports the library
explanation, but we cannot rule out graph capture as a contributing factor without
access to that machine.
Check the directory first because it costs nothing. If it is populated and you
still crash, try building without -DGGML_HIP_GRAPHS=ON.
Building, and what the fork is actually worth
Stock upstream llama.cpp works. We run a gfx906-specific fork with roughly sixty commits of kernel work on top, but it is worth being precise about what that buys: a median +6.1% prefill and +5.0% decode across 35 models, up to +40% decode on resident Q4_0. It is not the difference between running and crashing. That difference is the library above.
cmake .. -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx906 \
-DCMAKE_BUILD_TYPE=Release -DGGML_HIP_GRAPHS=ON
cmake --build . -j$(nproc)
If a host install fights you, mixa3607/llama.cpp:latest publishes daily
TheRock-based gfx906 images.
Verify before you download 15 GB
hipcc --version # HIP version: 7.16.26314-0000000
rocminfo | grep gfx906 # must appear as an agent
rocm-smi # confirms 16 GB, not 32
The lspci label lies. It reports “MI50 32GB” because
that is the generic PCI-ID string for the device, not what is soldered to the
board. Every sizing decision in this study assumes the 16 GB the
card actually has, check with rocm-smi, not with lspci.
Which quantisation, and why the obvious answer is wrong
Qwen3.8-27B is a dense 27.32 B DeltaNet hybrid (arch qwen35): 64 layers
of which 48 are linear attention and 16 full attention, plus one NextN/MTP block
= 65. Vocab 248 320, native context 262 144, thinking model, natively multimodal.
On 16 GB, three files are worth considering. We measured all three, fully resident where possible.
| Metric | Q3_K_M | Q4_K_S | Q4_K_M |
|---|---|---|---|
| File size | 12.57 GiB | 14.73 GiB | 15.66 GiB |
Decode tg64 -ngl 99 | 18.50 | 21.16 | 20.49 |
| Prefill pp512 | 183.20 | 226.57 | 223.15 |
| PPL16 (WikiText-2) | 7.7357 | 7.3826 | 7.3856 |
Context ceiling -ngl 99, f16 KV | 53 248 | 20 480 | 6 144 |
… with -ctk q8_0 -ctv q8_0 | 98 304 | ||
| Speculation (MTP) | +47.6% | +18.6% | OOM |
Q4_K_S beats Q4_K_M outright, faster on both axes, marginally better perplexity, and 0.93 GiB smaller. There is no reason to pick Q4_K_M on this card. The size difference also leaves room for the speculation context buffer, which Q4_K_M misses by 505 MiB.
Q3_K_M is not the cheap option, it is the long-context option. It is slower than Q4_K_S despite being smaller. Decode on this model is unpack-bound rather than bandwidth-bound on gfx906: Q3_K costs more to unpack than Q4_K, and on a card with 768 GB/s of real bandwidth the arithmetic loses before the memory bus does. Take Q3_K_M when you need more than 20 K of context, or the biggest speculation win on code.
Caveat, stated plainly. The comparison above was measured on
the Qwen3.8-27B-Uncensored line, whose files run ~2% smaller than
unsloth's dynamic quants. That 2% is decisive: unsloth's Q4_K_M does
not fit resident and needs -ngl 60, which costs 30% of
decode (14.46 against 20.49 tok/s).
We never tested an unsloth Q4_K_S. If you use unsloth files, verify the fit with a load test rather than trusting our ceiling numbers.
Running it
# everyday: fully resident, 16K context, ~21 tok/s
llama-server -m Qwen3.8-27B-Q4_K_S.gguf -ngl 99 -fa 1 -ctk q8_0 -ctv q8_0 -c 16384
# long context: switch model AND quantise the KV cache, do not offload
llama-server -m Qwen3.8-27B-Q3_K_M.gguf -ngl 99 -fa 1 -ctk q8_0 -ctv q8_0 -c 98304
# maximum speed on structured output (code, lists, JSON)
llama-mtp-spec -m Qwen3.8-27B-Q3_K_M.gguf -ngl 99 -fa 1 -c 4096 --temp 0Quantise the KV cache
Our own first campaign missed this. Every number in it was taken with the default f16 KV
cache; we never tried -ctk q8_0. It costs almost nothing and buys a great deal.
| Metric | f16 KV | q8_0 KV | Δ |
|---|---|---|---|
| Context ceiling | 53 248 | 98 304 | +84.6% |
| Context buffer at 53 248 | 3 477 MiB | 1 917 MiB | −44.9% |
| Free VRAM at 53 248 | 18 MiB | 1 567 MiB | +1 549 |
| Decode @ 0 / 8 K / 16 K | 18.57 / 17.74 / 17.16 | 18.39 / 17.57 / 16.92 | ≈ −1% |
| Prefill pp512 | 183.20 | 182.16 | −0.6% |
| PPL16 | 7.7352 | 7.7232 | within ±0.31 |
Nearly double the context, one percent of decode, no measurable quality cost. On a classical transformer the same switch is worth +59% decode on top (Qwen2.5-7B: 64.77 → 102.97 tok/s), because the gfx906 flash-attention kernel has a dedicated q8 tile path. Qwen3.8 sees only the memory half of that gain, as a DeltaNet hybrid, just 16 of its 64 layers hold KV at all, so the attention kernel is a small part of each step. The memory saving hangs on exactly those 16 layers and survives in full.
Do not reach for the turbo2/turbo3/turbo4
KV types on this build. They advertise 4.6× compression. Measured, all three cost
+222 to +232 MiB more VRAM than plain f16, and the overhead is the same
regardless of the advertised bit width, which is the signature of a fixed-size fp16 shadow
copy kept alongside the compressed cache.
turbo2, the most compressed at 2.5 bits per value, was both the largest and
the slowest: 35.6 tok/s against f16's 64.8. We found this by measuring peak VRAM, not by
reading the documentation, which claims the opposite.
What breaks, and what to do about it
Every one of these cost us hours or a wrong published number. They are listed in the order you are likely to hit them.
llama-completion hangs during warmup
This is the documented gfx906 SOLVE_TRI hang for hybrid SSM models, and it is real, but it affects exactly one of four tools, and not the one you serve with. Our own timings on Qwen3.8-27B Q4_K_S:
| Tool | with warmup | with --no-warmup |
|---|---|---|
llama-completion | hangs (>90 s, no progress) | works, 6 s |
llama-server | works | works |
llama-bench, llama-mtp-spec | works |
Our entire benchmark campaign ran past this unnoticed, because the benchmark binaries use different init paths. It retroactively explained a canary test we had killed an hour earlier without understanding why.
-ngl 99 is usually the wrong choice
It puts the output layer (241 MiB) on the GPU as well and halves your context
ceiling, on Q4_K_M from 10 240 down to 6 144 tokens, for a throughput
difference not worth measuring. -ngl 65 is the knee of the curve:
−5.8% for +67% context. Every layer below that costs about
3% each.
Do not trust llama-bench -r 3
It runs its repetitions back to back with no pause, so a power-capped card throttles inside the call. Our first round of Qwen3.8 numbers was wrong by up to 67% in exactly this way, and would have produced the headline “the fork is 33% slower than upstream”:
| Cell | -r 3 back-to-back | 3× -r 1, 240 s pauses | error |
|---|---|---|---|
| Q3 pp4096 | 108.55 ± 17.01 | 181.77 (spread 0.06%) | +67% |
| Q3 tg128 | 13.40 ± 4.45 | 18.61 (spread 0.22%) | +39% |
Run separate invocations, gate on junction temperature
(temp2_input < 45000), and report prefill and decode from
different runs. Every number in this study was taken that way.
Throttling is asymmetric, it flatters the faster arm
When you A/B two arms that generate the same token count, the slower arm spends longer under sustained load and therefore throttles harder. The comparison looks fair and is not. This artefact cost us a published claim of “+71% from speculation at 8 K depth”. Re-measured from equal cold starts, the honest figure is +13.5% on code and −3.7% on prose.
New rule: with unequal run times, compare equal wall-clock rather than equal token counts, and start both arms from the same gated temperature.
Two tool-specific landmines
llama-perplexity forces n_seq_max = 4, neither
-np 1 nor -c 512 changes it. On a DeltaNet hybrid that
quadruples the recurrent state buffer from 149.62 to 598.50 MiB,
enough to OOM a Q4 at -ngl 99 that runs fine under
llama-bench. Measure perplexity at -ngl 62.
llama-mtp-spec feeds the prompt unchunked; above 2048 tokens you
need -b <prompt length> or it asserts on n_batch.
What the hybrid actually buys, measured
The marketing line is “linear attention scales better”. The measured version is less dramatic at short context and more impressive at long.
At 4096 tokens the prefill advantage over a comparable full-attention model is only 3.6 percentage points of falloff (−2.2% against −5.8%), at that length the quadratic term is not yet dominant. The real win is the KV cache: 128 MiB at 4096 tokens against roughly 1 GiB for the similarly-sized Qwen3-32B, a factor of about eight, because only the 16 full-attention layers hold KV at all.
That compounds with offload. Lowering -ngl buys twice, fewer weights
and less KV on the card. The context buffer shrinks from 64 MiB/K at
-ngl 65 to 48 MiB/K at -ngl 50. Rule of thumb:
every 5 layers moved to the host roughly doubles the context ceiling
(10 240 → 28 672 → 53 248 → 81 920).
Q3_K_M, -ngl 99, separate cold-started runs, median of three, spread
below 0.3%. Total falloff from 0 to 48 K: −18.3%, where a
classical transformer of this size loses a multiple of that.
Every number we recorded
All figures: 1× MI50 16 GB, gfx906, ROCm 10.1, 150 W cap, -fa 1,
separate prefill and decode invocations, cold start gated on junction temperature
below 45 °C, discard run first, median of three. Model line is
Qwen3.8-27B-Uncensored unless noted.
| Quant | File | pp512 | pp4096 | tg64 | tg128 | PPL16 |
|---|---|---|---|---|---|---|
| Q3_K_M | 12.57 GiB | 183.20 ± 0.27 | 179.39 ± 0.15 | 18.38 ± 0.03 | 18.39 ± 0.04 | 7.7357 |
| Q4_K_S | 14.73 GiB | 226.57 ± 0.27 | 21.16 | 7.3826 | ||
| Q4_K_M | 15.66 GiB | 223.15 ± 0.04 | 218.05 ± 0.23 | 20.49 ± 0.04 | 20.45 ± 0.04 | 7.3856 |
| Setup | 0 | 4 096 | 8 192 | 16 384 | 32 768 | 49 152 |
|---|---|---|---|---|---|---|
Q3_K_M -ngl 99 | 18.50 | 18.04 | 17.66 | 17.10 | 16.03 | 15.11 |
Q4_K_S -ngl 99 | 21.16 | 20.11 | ||||
Q4_K_M -ngl 60 | 14.81 | 14.07 | 13.69 | |||
Q4_K_M -ngl 55 | 11.89 | 10.96 | 9.81 |
The most useful line in this study. If you want large context, take the resident Q3, not an offloaded Q4. Both reach 53 248 tokens, but at 32 K depth the Q3 delivers 16.03 tok/s against the offloaded Q4's 9.81, a 63% difference. Offloading to buy context is a worse trade than dropping a quantisation level to keep the model resident.
| Depth | 0 | 8 192 | 32 768 |
|---|---|---|---|
| pp2048 | 183.6 | 170.5 (−7.1%) | 140.9 (−23.3%) |
At 32 K depth the hybrid still prefills faster than Qwen3-32B does at zero depth (138.5).
| -c | 8 192 | 16 384 | 32 768 | 40 960 | 49 152 | 51 200 | 53 248 | 55 296 |
|---|---|---|---|---|---|---|---|---|
| free MiB | 2 844 | 2 332 | 1 308 | 796 | 284 | 156 | 28 | ✗ |
| -ngl | 4 096 | 8 192 | 16 384 | 32 768 | 49 152 | 65 536 | ceiling |
|---|---|---|---|---|---|---|---|
| 99 | 152 | ✗ | ✗ | ✗ | ✗ | ✗ | 6 144 |
| 65 | 406 | 150 | ✗ | ✗ | ✗ | ✗ | 10 240 |
| 60 | 1 618 | 1 378 | 898 | ✗ | ✗ | ✗ | 28 672 |
| 55 | 2 763 | 2 539 | 2 091 | 1 195 | 299 | ✗ | 53 248 |
| 50 | 3 856 | 3 664 | 3 280 | 2 512 | 1 744 | 900 | 81 920 |
The two failure modes are distinct and worth knowing apart: 55 296 fails on the constant 495 MiB compute buffer, 65 536 fails on a 4 GiB KV block allocation. The context buffer grows at 64 MiB per 1 K tokens, twice as fast as extrapolating from the 128 MiB@4K KV figure suggests. Measure, do not extrapolate: our own 64 K prediction was off by a factor of two.
Q4_K_M, tg64 at depth 0. Offload costs a fixed amount rather than a steeper curve:
falloff over 16 K at -ngl 60 is −7.6%, identical to the resident Q3.
Moving layers to the host shifts the level, it does not change the shape.
Speculative decoding via the model's own NextN head
Qwen3.8 ships an MTP block as layer 64. Unlike draft-model speculation, which we measured at +0.9% and abandoned, this costs no second model and no second forward pass.
| Model | off | on | gain | acceptance | tok/step |
|---|---|---|---|---|---|
| Q3_K_M | 18.52 | 27.34 | +47.6% | 100.0% (49/49) | 2.02 |
| Q4_K_S | 21.10 | 25.02 | +18.6% | 96.8% | |
Q4_K_M -ngl 99 | OOM (505 MiB short) | ||||
Q4_K_M -ngl 60 | 14.71 | 11.46 | −22% | 96.8% |
Two conditions must hold simultaneously for speculation to pay: the NextN block
must be resident, and there must be room for the second context.
At -ngl 60 the draft block itself sits on the host, so every draft
crosses PCIe, which is why the arm loses 22% despite 96.8% acceptance.
That is a VRAM question, not a flaw in the speculation path.
| Depth | Task | off | on | gain | acceptance |
|---|---|---|---|---|---|
| 4 096 | code | 18.56 | 25.78 | +38.9% | 95.0% |
| 8 192 | code | 17.80 | 20.20 | +13.5% | 91.0% |
| 8 192 | free prose | 17.78 | 17.12 | −3.7% | 71.2% |
The MTP context cost rises with depth (4.40 → 11.09 ms/step) while the saving stays flat. Use speculation for structured output; turn it off for prose.
| Quant | Test | Fork | Upstream | Δ |
|---|---|---|---|---|
| Q3_K_M resident | pp512 | 185.85 | 169.33 | +9.8% |
| Q3_K_M resident | pp4096 | 181.77 | 173.03 | +5.1% |
Q4_K_M -ngl 60 | pp512 | 206.98 | 150.46 | +37.6% |
Q4_K_M -ngl 60 | pp4096 | 202.70 | 186.78 | +8.5% |
| Q3_K_M resident | tg64 | 18.54 | 18.14 | +2.2% |
Q4_K_M -ngl 60 | tg64 | 14.57 | 14.13 | +3.1% |
Prefill (the tiled MMQ kernels) benefits; decode (the matvec path) barely does. That split is consistent across our whole 36-model roster, on resident Q4_0 models it reverses and decode gains up to 40%.
| Test | Q3_K_M | Q4_K_S / Q4_K_M |
|---|---|---|
| Canary (“capital of France, one word”) | ✓ Paris, clean <think> | ✓ Paris |
| Code (interval merge) | ✓ correct, O(n log n) | ✓ correct, also avoids mutating input |
| 16 K recall (70 KB WikiText) | ✓ correct | |
| Vision (mmproj-F16, red circle) | ✓ described correctly | |
| Degeneration / repetition | none | none |
| Use case | Configuration | Expect |
|---|---|---|
| General purpose | Q4_K_S -ngl 99 -ctk q8_0 -ctv q8_0 -c 16384 | 21.2 tok/s, best quality |
| General + code | Q4_K_S -ngl 99 + MTP | 25.0 tok/s |
| Q4 quality, more context | Q4_K_S -ngl 60 -c 32768 | 32 K context |
| Code maximum | Q3_K_M -ngl 99 + MTP | 27.3 tok/s (quality −4.6%) |
| Maximum context | Q3_K_M -ngl 99 -ctk q8_0 -ctv q8_0 -c 98304 | 98 K context, ~17 tok/s |
| Q4_K_M | superseded by Q4_K_S in every respect |
What “supported” turns out to mean
It would be convenient to frame this as “our fork can do what stock cannot”. That is not true, and the numbers above say so plainly: the fork is worth single-digit percentages on this model. The decisive difference is not any of our sixty commits, it is a fully built math library. Install the same ROCm stack and stock llama.cpp will probably get you there too, just slower.
What the case does show is how much load the word supported carries. The MI50 has been deprecated for years. From the perspective of those two bug reports it behaves like it: models crash, nobody answers, the reports sit unconfirmed. From ours, the same card runs a 27 B model at 53 K context and a 117 B mixture-of-experts at 32 tok/s.
Both observations are correct. What separates them is a package source and knowing which flag to set.
The largest single throughput win of that week was not an optimised kernel either. It was switching off an inherited kernel that turned out to be 5.2× slower than the library function it was meant to replace. And the most useful finding of the day was that a crash does not mean a thing cannot be done. Sometimes a file is simply missing.