The gap between one and many
Part 1 optimized a single token; Part 2 turned one into two. Part 3 covers the space between them: two to sixteen columns, a range neither the matvec nor the matmul kernel was built for, and a Rust server that produces this range in practice. The main kernel result is a demonstrated “cannot be done.” The server result: it beats the original at every level of concurrency.
the ladder that established the floor
versus the previous path
tokens/s at 4 concurrent requests
alongside three running streams
Why a few columns are no man's land
GPU inference has two well-understood worlds. Generating a single token uses a matveckernel: matrix times vector, memory-bound, optimized over months in this series. Processing entire prompts uses a matmultiled kernel: matrix times matrix, compute-bound, with large tiles. Between them lies no man's land: two to sixteen columns, produced by speculative decoding, batch verification, and small-scale serving. Matvec pays almost the full price per column; the tiled kernel measures twice as expensive.
Part 2 ended with exactly this diagnosis: after every fix, the marginal second column still cost 20–31%, capping speculation. The task for Part 3: a mini-GEMM for that gap. Alongside it, the server side that produces these batch sizes in daily operation: fork-serve, a slim Rust server on top of the fork, was to learn continuous batching.
The main result is a demonstrated “cannot be done”
2–4 columns: a structural floor
Fifteen kernel variants competed on nine real matrix shapes: column splits, row splits, LDS staging in several geometries, more rows per workgroup, all with register and occupancy data per variant. At two to four columns, not one beat the existing path across all matrix widths. One won on exactly one shape and lost on all the others. The gating lesson from Part 1 removed it before it could cause damage.
This establishes something more valuable than a gain: on this architecture, the ~20–30% cost of the second column is the floor. Each column inevitably requires eight dot-product instructions plus activation loads in the same execution stream. Going below that requires changing the activation format from quantization onward, or tensor cores that this chip does not have. Future optimization attempts here can refer to the variant ladder and save themselves a week.
Cooperation wins from five columns onward
The picture changes at five columns. The winning variant (“csw”) has two or four waves share the columns while the weight blocks in each section are cooperatively loaded into LDS once and reused by all waves, arranged without bank conflicts: a lesson from Part 1's megakernel prototype.
The end-to-end effect, independently remeasured: a 9B model, eight parallel sequences, decode 143.6 → 155.2 tokens/s (+8.0%); at twelve sequences, −17% step time. And a subtlety only the server measurement revealed: real operation mixes decode rows and chunks of new prompts into exactly the batch sizes mini-GEMM targets: +6.7% at four concurrent requests, even though the pure four-sequence benchmark is unchanged.
fork-serve learns continuous batching
fork-serve is the in-house Rust server on top of the fork: handwritten FFI bindings with machine-verified memory layout, RAII around every resource, and SSE streaming without measurable overhead. Until now, it handled requests one after another. Now up to eight requests share every decode step. A scheduler builds a common batch per step: ongoing decodes first, then chunks of incoming prompts within a fixed token budget so nobody blocks. Requests enter and leave between steps.
| Concurrent requests | fork-serve | llama-server (HTTP) |
|---|---|---|
| 1 | 110.3 | 113.2 |
| 4 | 240.8 | 233.3 |
| 8 | 307.6 | 295.5 |
Corrected version: An earlier measurement in this project put llama-server at just 169.5 tokens/s with four requests and inferred a 46% lead for fork-serve. Remeasurement on 16 August found llama-server at 233.3: the old value was an outlier, and the 46% claim is withdrawn. The actual lead is 3–4% from four concurrent requests onward, while fork-serve is slightly behind for a single user. The defensible conclusion remains: the custom serving layer has almost no measurable cost. The real practical benefit of continuous batching appears in another number anyway: a request joining alongside three running streams gets its first token after 50 milliseconds and then streams at a stable 22.5 ms per token.
A revised verdict, with a defined scope
Part 1 disabled HIP graphs on this architecture: slower single-token decode and a documented memory leak in the ROCm runtime. Multi-sequence operation is different: kernel launches dominate, and graphs provide +38%. The leak? 360 requests in waves of four with constant sequence changes, the exact historical trigger pattern, measured independently twice: after the warmup plateau, +104 kB and then +12 kB per 120 requests, with no trend. The default now depends on operating mode: on for batching, off for single-token decode. Verdicts have scopes.
The old measurement was not wrong; it covered a different operating pointThe ABI check catches real drift
When extending the bindings, the machine-checked structure comparison reported a mismatch: the fork had added a field, leaving the Rust side eight bytes too small. Without the check, this would have silently corrupted memory; with it, the correction happened at build time. Every new structure is now added to the check.
Discipline pays off precisely when nobody is lookingThe MTP port proves itself against the reference
The speculative logic from Part 2 was ported to Rust as a separate server mode, deliberately not combined with continuous batching. Draft rows change the batch composition for every sequence; that design is documented rather than half-implemented. The proof: identical counters and identical text to the C++ tool, 83.7 tokens/s.
A port is not finished until its counters matchStability in numbers, not adjectives
400 mixed-length requests over ten minutes: zero errors, flat memory use after warmup, clean shutdown in both modes. The four-prompt test, in which each answer must belong to its question, rules out the classic batching error where sequences contaminate one another.
Cross-talk is the bug you find before the first user, or neverWhat Part 3 adds to the series
The kernel work provides a complete map: from batch 1 to the matmul world, every range now has its measured best kernel, every boundary its evidence for each matrix width and quantization type, and the range with nothing left to gain its proof. The server side turns this into everyday value: more throughput than the original at every level of concurrency, with latency that feels like single-user operation.
And the series' methodology gains three entries: A proven negative result can replace a design plan: the 15×9 ladder saves every successor a week. Verdicts have scopes: the graphs decision was right for its original operating point and wrong for the new one; preserving a verdict without recording its operating point preserves mistakes. And verification should happen twice: every central figure in this part was independently remeasured, including a failed attempt caused by the checking script itself and recorded exactly that way in the log.
Three parts, one pattern: most of the performance never came from one heroic idea, but from removing precisely measured obstacles. Every answered question identified the next one.