Taking stock
After months of kernel work, the uncomfortable question: what is it all worth? 36 models, both builds, identical conditions. The short answer, median +5%, is the least useful number in the entire series. The range runs from −24% to +676%, and a 27B model released that very day shows why the question was framed incorrectly. By the end, one of our own claims had to be withdrawn.
the series’ most misleading number
the work actually targeted
(prefill, APEX 35B Q2_K)
wins, reported honestly
Why a median lies here
Across 35 comparable models, the fork delivers a median +6.1% prefill and +5.0% decode. As a headline, that would be sobering and entirely misleading. The optimizations in this series were never general-purpose: they targeted specific quantization formats, specific kernels, and specific operating points. A median averages away that very precision.
Sorting the same measurements by format and memory placement turns one meaningless number into a map, and that map confirms, point by point, what the previous four parts had claimed:
Bar = observed range, marker = median. Zero line at left. 35 models, two runs per build with cooldowns.
| Class | n | Median Δ prefill | Median Δ decode | Decode range |
|---|---|---|---|---|
| Q4_0/Q8_0/MXFP4 · Resident | 15 | +2.0% | +15.0% | −0.1 … +39.8% |
| Q4_0/Q8_0/MXFP4 · Offload | 8 | +8.3% | +4.8% | +0.9 … +6.8% |
| K-quant · Resident | 7 | +6.1% | +5.0% | +1.2 … +8.5% |
| K-quant · Offload | 5 | +11.3% | +3.1% | +0.3 … +8.4% |
This table contains three findings, all three predicted:
First, the division of labor. The decode gain sits in resident Q4_0 (+15% median, peak +39.8%), exactly the class targeted by the matvec work in Parts 1 and 3. Conversely, the prefill gain sits in K-quants (+6.1 and +11.3%, versus only +2.0% for resident Q4_0), reflecting the most recent tiling work. In other words, we fixed Q4_0 decode and K-quant prefill, and the measurements cleanly separate the two.
Second, the boundary of our influence. Models largely resident in system RAM gain little in decode: the PCIe path decides performance there, not our code. Llama-3.3-70B, three-quarters offloaded, is practically at vanilla performance with +1.2%. That is not a failure, but a clear boundary: this is where our influence ends.
Third, the upward outlier. APEX 35B gains +676% in prefill (121 → 939 tokens/s). This is real and explained: the model uses a format whose vanilla kernels suffered register spills. It was the very first finding in this series, now visible directly against baseline for the first time.
A 27B that gains almost nothing from the kernels, yet gains the most overall
Qwen3.8-27B was released on the day the measurement campaign ran. A dense 27-billion-parameter model that fits in 16 GB of VRAM, with a hybrid architecture inside: 48 of its 64 layers use linear attention, only 16 use conventional attention. For this project it was the ideal test case, and it exposes kernel percentages as the wrong measure.
The starting point 18.32
60 optimization commits … for +1.7% 18.63
The built-in predictor, without a single line of new code 25.65
Versus 17.80 without speculation: the gain shrinks with depth 20.20
At the kernel level, this model is a minor beneficiary: +1.7% decode, +4.4% prefill. No coincidence: it uses a K-quant format, and K-quants do not participate in the matvec optimizations. Judging the work by kernel percentages would make it “barely worth mentioning.”
At the system level, it is the roster’s biggest winner: because the architecture had already been ported, the speculative path worked immediately, with 95% acceptance and +38.9% in the short window, with identical output. At 8K context, this falls to +13.5% (20.20 versus 17.80), and only for structured output: on free-form prose continuation, acceptance falls to 71%, and speculation loses 3.7%. An earlier version of this study reported +71% here. That figure was a throttling artifact and has been withdrawn; see the box below.
The lesson extends beyond this model: the value of optimization work is not confined to its kernel percentages. Architecture support, a speculative path, a server layer: these are multipliers absent from any kernel measurement, and here they account for the lion’s share.
Three losses and a withdrawn claim
Vanilla wins
Three models run slower on the fork than on vanilla, all in prefill, all small and fully resident in GPU memory:
| Model | Vanilla | Fork | Δ | Decode Δ |
|---|---|---|---|---|
| BitNet-b1.58-xl | 2636.2 | 1989.8 | −24.5% | −0.1% |
| Gemma-4-E2B QAT | 2756.3 | 2451.4 | −11.1% | +0.8% |
| Gemma-4-E4B QAT | 1433.2 | 1358.8 | −5.2% | +3.3% |
The common denominator is clear: small, resident models with unusual geometry. Decode is unremarkable in all three cases, so the regression is clearly in the tiled kernel. The obvious hypothesis, that our special rule for narrow tiles on small compute grids misfires here, or that vanilla contains newer code than our base, is uninvestigated, but testable in half an hour: flip the switch and measure. That is how it appears in the record, rather than being explained away in a footnote.
And then our own number failed to hold up
Parts 3 and 4 of this series contained a striking claim: with four concurrent requests, our Rust server delivered 46% more than llama-server because the latter “loses a quarter of its potential in its own serving layer.” Remeasurement during this assessment finds llama-server at 233.3 rather than 169.5 tokens/s: the earlier value was an outlier.
| Concurrent requests | llama-server | fork-serve | Δ |
|---|---|---|---|
| 1 | 113.2 | 110.3 | −2.6% |
| 4 | 233.3 | 240.8 | +3.2% |
| 8 | 295.5 | 307.6 | +4.1% |
Withdrawn: “llama-server loses a quarter of its potential in its own HTTP layer” and the resulting +46%. The actual lead is 3–4% from four concurrent requests onward; in single-request operation, our server trails slightly. What remains valid: the server layer measurably costs almost nothing, and the practical value of continuous batching is in a different number anyway: 50 milliseconds to first token while three other streams are running. Both case studies have been corrected.
And a second one, found while writing this text
A question about the 27B model’s 8K result exposed a contradiction in our own data: the context ladder measured 17.81 tokens/s without speculation, while the speculative test for the same state measured only 12.54. Remeasuring from a cool start resolves the discrepancy and costs us the second headline.
| Task | Without MTP | With MTP | Δ | Acceptance |
|---|---|---|---|---|
| Code task | 17.80 | 20.20 | +13.5% | 91.0% |
| Free-form prose | 17.78 | 17.12 | −3.7% | 71.2% |
| Previous figure | 12.54 | 21.47 | +71.2% | 98.4% |
Withdrawn: “+71% from speculation at 8K depth” and the claim that the gain grows with depth. The cause is a type of trap this series had not encountered before: after the 53-second prefill, both arms generate the same number of tokens, but the slower arm takes longer, spends longer under sustained load, and throttles more. Throttling acts asymmetrically in favor of the faster arm and inflates the gain. The earlier report considered the A/B fair “because both arms prefill identically.” That holds for prefill, not generation. In reality, the gain shrinks with depth (+38.9% → +13.5%) because the prediction context becomes more expensive. New rule: for A/B tests with unequal runtimes, start both arms equally cool, and for long runs compare equal wall-clock durations rather than equal token counts.
A secondary finding from the same series that would otherwise never have made it into a report: our server collapses at the slot boundary. At exactly eight concurrent requests and eight configured slots, throughput falls to 229.9; with sixteen slots, it is 307.6. Slot count needs headroom above expected concurrency.
What is left in the ledger
For the model class the work targeted, dense models in 4-bit block formats fully resident in GPU memory, the fork delivers +15% median decode, up to +40% in individual cases. For everything else, gains are in the single digits; for models mostly resident in system RAM, practically nothing. This is not disappointing, but precisely what targeted optimization can deliver: a lot where it acts, little elsewhere.
The real return, however, is not in that column. It is in what became possible: a 27B model with 48K context on a 16-GB card, a speculative path that speeds up code output by 39%, a 117-billion-parameter model running at 32 tokens/s, eight concurrent users with single-user latency, and card behavior now measured well enough to predict what a model will achieve before every download.
Five parts, one consistent method: measure rather than assume, document negative results on equal terms, and cross-check every number, including our own, especially the flattering ones. This part found three losses and a withdrawn claim. That is exactly why you take stock.