The kernel that waited for itself

A dip at sixteen that nobody had measured

The fork was exceptionally well tuned for a single user: MTP speculation, anchor caching, around 100 tokens/s on an 8B model. What happens with many concurrent sequences had never been measured systematically. A comparison against vLLM (campaign AX) provided the occasion and immediately two findings: the server’s batch scheduler assigned sequence IDs as a LIFO stack, causing llama.cpp with split-KV to divide each decode step into two or three substeps (throughput erratically dropping to one third). And the engine itself had a dip at exactly 16 sequences: B12 354, B16 328, B24 397 tokens/s.

The dip had a simple cause (campaign AY): from 13 columns onward, Q4_0 fell back from the fork’s own mini-GEMM kernel to the generic MMQ path, an old default whose justification no longer held. A switch delivered +17%. But then came the real wall: the profiler showed that the mini-GEMM kernel accounted for 76% of the decode step, and the compiler’s resource report explained why: 105–117 vector registers per thread, allowing only 2 waves per SIMD. Nine tile variants and forced occupancy changed nothing. The kernel had reached the limits of its design.

Seven variants, one microbenchmark, many mistakes

A standalone HIP harness (scratchpad/m12-lds-gemm/bench.hip) with Q4_0/Q8_1 quantization, a CPU reference, and a copy of the production kernel (“csw”) as baseline made every variant measurable in seconds across the four real matrix shapes of an 8B model. Times here: microseconds per invocation, q/o projection 4096×4096.

Three ways hipcc forces a pipeline back into lockstep

The fact that prefetch (V3) was slower fit no theory, until the ISA was available (--save-temps, then a trace of only the relevant instructions: G = global load, v = wait for loads, W/r = LDS write/read, d = dp4a, | = barrier, ^ = branch). Three findings, three countermeasures.

Finding 1, Loads sink into branches

G2 v2 W3 ^ G3 v2 W2 v W |      ← Staging loop with if (b < nblk)

Two loads, wait, three loads, wait. A condition in staging split the basic block, and the compiler moved every load to where it was needed, immediately before use. One latency became three. Countermeasure: straight-line staging without if (the dispatch guarantees full chunks) and asm volatile("" ::: "memory") immediately after the loads.

Finding 2, Stores move ahead of compute

^2 G4 r28 ^2 v W v W2 v W2 ^ | ... d128      ← Double buffering, first attempt

The loads for chunk c+1 were at the front, but so were the LDS stores that must wait for them; the dp4a block came afterward. The store does not depend on compute, so it was allowed to move upward. A memory barrier alone did not help: arithmetic is not a memory operation and moves freely across it.

Finding 3, The countermeasure must bind values, not memory

G4 r5 d6 r d2 r d8 r3 d16 ... d16 ^2 v W v W2 v W2 ^ |      ← Target reached

for r, j: asm volatile("" : "+v"(acc[r][j]) :: "memory");   // before the stores

Using the accumulators as input/output operands of an empty asm block forces the compiler to finish all compute before that point; the memory clause keeps stores behind it. Only then were loads, computation, and waiting in the order intended by the design, and V7 became the fastest variant.

db-gemm in one paragraph

Four models against baseline

llama-batched-bench, flash attention enabled, f16 KV, 64-token prompt, 128 generated tokens, with 40–60-second pauses between runs. Baseline = the same build with GFX906_DISABLE_DB_GEMM=1, thus including all earlier improvements. B1 and B8 are unchanged; this kernel does not touch those paths.

Through the server and against vLLM

Through fork-serve (batch scheduler, 32 slots, q8_0 KV), the engine gains translate to: B13 299 → 357, B16 332 → 451 tokens/s. For comparison, the same test against vLLM (gfx906 fork, AWQ): B16 329, B32 383. The decode step at 16 streams fell from 42.6 to 29.0 ms; the kernel’s share from approximately 32 to approximately 18 ms.

What correctness was measured against

Reference in the harness: CPU calculation on quantized values, relative tolerance 2·10⁻³, every variant, every shape. Column symmetry: 16 identical activation columns produce bit-identical output columns (csw and db-gemm). test-backend-ops: twelve new large MUL_MAT cases (1024–12288 rows, 13/15/16/24/32 columns, K 4096/12288, Q4_0 and Q8_0), demonstrably using the new path through debug tracing, all OK; the full run reports 0 FAIL. Model: 16 identical greedy sequences produce two text variants rather than one with db-gemm, at a genuine near tie (114 of 125 characters shared), not within matmul. Summation order differs from the old kernel, which may break close logit ties differently.

Coverage evidence instead of assumption: GFX906_MINI_GEMM_DEBUG=1 logs every dispatch with shape, column count, split-K, and groups. On Qwen3-8B, all seven matmul shapes per layer use the new kernel.

The lever list: five gains, three honest zeros

After the kernel, a list of seven further levers remained, ordered by value. All were investigated; the results show how the method works even where the final outcome is no gain.

The long-context finding: an occupancy trap in attention

All earlier figures use short context. At 32k tokens, the picture changes: prefill attention accounts for 55% of the step. The fork-serve default (q8_0 KV) selected a gfx906 kernel tuned for decode that ran at only half the generic f16 kernel’s speed during prefill (246 versus 485 t/s @32k). The cause was the same occupancy wall as in Section 1, but reversed. Head-dimension-128 configurations for 32/64 query columns ran at target occupancy 2, capping registers and throttling a kernel that wanted more. Occupancy 2 → 1: Prefill +40…48% (16k 397 → 556, 32k 246 → 363 t/s), decode unchanged, 3344/3344 tests green. The lesson mirrors the decode kernel: occupancy is a two-way control. Forcing it up chokes a latency-bound kernel just as forcing it down starves a bandwidth-bound one. Only measurement in both directions reveals which case applies.

The cheapest lever was in the driver, not the code

After weeks of kernel work, it emerged that the card had been running at 150 W all along, although its PowerPlay table permits 190 W, set by a systemd service we had created months earlier. Raising the cap: pp512 +10%, B16 decode +7%, 32k decode +5.7%, in five minutes. Under sustained load, the clock now pins at 1700 MHz at 166–169 W and 75 °C junction. The card no longer even reaches its new limit; it runs at the VBIOS clock ceiling. Memory clock, however, could not be increased: the SMU rejects any state above approximately 1090 MHz and falls back to 800 under load, the read path saturates at approximately 890 GB/s at the fabric anyway, and merely uploading the PowerPlay table costs 1.5–2.5%. We noticed only because we uploaded the unchanged table as a control.

Against current upstream, and the one place it was better

A fresh build of official llama.cpp (b10524) on the same card, with the same models: the fork leads by +20 to +78%, and in long-context decode by a factor of 4. At 32k context, upstream loses 37% of its decode speed, the fork 5%. Exactly one measurement went to upstream: 32k prefill with q8 KV, 489 versus 376 t/s.

The cause was not a better kernel, but a better dispatch decision. Upstream converts the q8 cache to f16 once per attention invocation and runs the f16 kernel: correct for prefill (many columns amortize conversion), catastrophic for decode (14.9 t/s: the entire cache converted per token). We had the mirror image. Combining both, specialized kernel up to 32 columns, conversion above that, yields Prefill 4k/16k/32k: 1050 / 726 / 514 t/s (previously 903/536/376), with decode unchanged. Twenty lines of dispatch logic, +37% at 32k. The lesson: a routing table deserves the same care as the inner loop.

Where kernel work ends and policy begins

Eventually the instruction sequences were tight, and the remaining levers were no longer instructions, but per-request decisions. The clearest example is speculative decoding: a draft head guesses the next token, and the main computation verifies it. Whether it pays off is the product of two measurable quantities: acceptance (code and repetition 90–100%, free prose 49–79%) and the marginal cost of a verification row. For dense, resident models, weights are already in VRAM, so another row costs almost nothing: +15 to +40% across all prompt types. For the MoE model with CPU experts, every row triggers transfers at 14 GB/s, and a rejected draft has a real cost: the same speculation loses 23% on prose there.

Because this can change within a session, the server now decides. A window over the last 16 verified drafts disables speculation when acceptance falls below 70%, with exponentially growing holdoff. Continuous prose is rarely reprobed, but a switch to code is detected. The real trick was not the window, but what gets suspended: suppressing only the verification row recovered one third of the loss (66.9 → 72.1 t/s); only deferring the head pass, collecting pairs and feeding them back in one batch when the gate opens, reached 84.3 t/s, versus 87.1 without speculation.

Multi-stream operation follows different economics, and overlooking that cost us a day: the head runs for the other streams anyway, so a draft costs only its verification row. The 70% threshold calibrated for solo operation therefore blocked streams whose drafts would have been almost free, losing 23% on a good prompt. The solution was to remove the constant entirely: two moving averages of nanoseconds per token, with and without a draft, plus periodic cross-checks. Multi-stream prose: 110 → 142 t/s.

Three measurement traps that produced convincing wrong numbers

While building this controller, we fell into every available trap, and each produced plausible figures.

The cost window was too narrow. The first version measured only the head phase, but draft rows are paid for in the main decode. The controller immediately made the wrong decision (dense/prose: 33.5 instead of 46 t/s), and the number looked entirely normal.

Deferred work landed in the wrong bucket. With the gate closed, head pairs accumulated; the later batch decode was charged to draft mode, poisoning the very comparison the decision relied on (dense/repetition: 60.4 instead of 85.7).

And the most expensive: an A/B test with a fresh process per arm measures cold start, not performance, for an adaptive component. We had measured with clean alternation, a cooling gate, and two runs per configuration, and obtained +1.7%. Only six consecutive runs in the same process revealed what really happens: from run two, acceptance drops to 0.0% (the controller has decided), and throughput rises from 117 to 145 t/s. The real gain was +29%, not 1.7%: the controller had never progressed beyond learning during measurement. We then shortened its time constant (first cross-check after 48 rather than 240 steps) so it also takes effect in short sessions.

What the card delivers today

To finish, all models together, at the 190 W cap, with a cooling gate before every measurement:

Two operating classes stand out. Dense models run at the limit: 170–190 W, 93–99% utilization, 85–92 °C junction. MoE models with CPU experts stay at 104–168 W and 59–78 °C; there the GPU waits for the bus. And: utilization is not a throughput metric. Lightning delivers 147 t/s at 76% utilization, while dense 27B delivers only 82 at 96%. The MoE computes with 3 billion active parameters, the dense model with 27. For the MoE models, a second card would therefore be the direct lever, not a larger power budget.

What remains open

Status after the second round: Of eight mapped levers, five are realized (power cap, FA prefill, FA decode, small-kernel chain, q8 prefill dispatch), two closed with evidence (MMQ redesign is compute-bound, memory-clock overclocking has three separate limits), and the W4A4 question is answered rather than implemented: the core runs 1.76 × faster, while quality at +0.81% perplexity sits exactly at our tolerance boundary. Remaining:

The VALU floor. In the finished kernel, pure arithmetic is the largest item at approximately 20 of 47 µs (4096×4096): nibble unpacking and block scaling double the instruction count relative to dp4a operations alone. A further step would require a different weight format or instruction-level tricks.

The second card. The full measurement turned it from a wish list item into an evidence-based recommendation: MoE models run at 54–76% GPU utilization and 104–168 W because the GPU waits for CPU experts. More VRAM means less offload, their direct lever rather than a larger power budget. Dense models are the opposite: they sit at 99% and 190 W.

MoE and K-quants. Both paths are built and tested, but deliver little here: K-quants because the existing tiled baseline is strong, MoE because some experts execute on CPU on a 16-GB card.

Only the batch path. Nothing changes at 1–12 sequences; the fork was already at its floor there (M4/M9/M10). Someone working alone with a model will notice nothing from this campaign. Anyone planning subagents, multiple users, or a multi-card setup gets up to 1.6× more from the same card.

Reproduction

hipcc --offload-arch=gfx906 -O3 -std=c++17 scratchpad/m12-lds-gemm/bench.hip -o bench && ./bench 200
./build/bin/llama-batched-bench -m MODEL.gguf -c 8192 -b 2048 -ub 512 -npp 64 -ntg 128 -npl 1,8,13,16,24,32 -ngl 99 -fa on
GFX906_DISABLE_DB_GEMM=1 ./build/bin/llama-batched-bench ...                     # Baseline
GFX906_MINI_GEMM_DEBUG=1 ./build/bin/test-backend-ops -o MUL_MAT -b ROCm0         # Abdeckung + Numerik