Fourteen hours to 100 tokens per second
A 35-billion-parameter model with 256 experts, two graphics cards from 2018, and a target that seemed unreachable according to our own model calculations. How one kernel trace overturned the premise, which eight levers then worked, which four did not, and why the final ten percent is no longer a kernel problem.
Starting point
At noon on September 2, Qwen3.6-35B-A3B ran in the fork at 72 tokens per second without speculation. With the DFlash drafter, it reached 135 on code, but only 63 on prose and 37 at six thousand tokens of context. The model is a mixture of experts: 40 layers, including 30 recurrent Gated DeltaNet layers and 10 attention layers. Each token computes 8 of 256 experts plus one shared expert, around two gigabytes of active weights.
The project’s model calculation predicted roughly 5.7 milliseconds per token for two gigabytes at one terabyte per second of memory bandwidth. Measurement showed 14.7. Nine milliseconds were unexplained, and the working hypothesis was that the card was bandwidth-bound, so the route to 100 went through the drafter. The campaign began with three hypotheses and a commitment to measure each before writing any code.
The trace that changed everything
The first tool was not a kernel, but a rocprofv3 kernel trace over 17 decode tokens. It showed 1452 kernel launches per token at 13.9 milliseconds of pure GPU time, versus 13.5 milliseconds wall-clock. The card spent practically all its time in kernels, and effective bandwidth was 145 gigabytes per second, one sixth of what was possible.
| Kernel class (start of DX) | Launches per token | ms per token | Share |
|---|---|---|---|
| Expert matvecs (mul_mat_id) | 241 | 4.10 | 29.5% |
| Recurrent-layer state gather (get_rows) | 62 | 2.07 | 14.9% |
| Dense matvecs (warp_coop Q4_0) | 70 | 1.01 | 7.3% |
| Broadcast arithmetic (bin_bcast) | 191 | 0.90 | 6.5% |
| Activation quantization (quantize_q8_1) | 200 | 0.89 | 6.4% |
| F32 router (mul_mat_vec_f) | 80 | 0.79 | 5.7% |
| Top-k routing | 40 | 0.58 | 4.1% |
Kernel trace of one decode token, September 2 at noon. All classes sum to 13.9 ms; “glue” at the launch floor (kernels below 5 µs) alone accounts for 3.3 ms.
Three conclusions followed. First: the baseline is kernel- and launch-bound, not bandwidth-bound. Every kernel below five microseconds costs more to launch than to execute. Second: an eight-row batch costs only 2.2 times as much as one row, because additional rows are almost free when bandwidth is unused. A single benchmark measurement therefore disposed of the hypothesis that the two cards could pipeline verification. Third: 2.07 milliseconds per token went into gathering recurrent state, even though for a single token the row being read is the same row written back.
Eight levers that worked
The trace determined the order: large items first, then glue, and finally the copy between cards. Every change was measured with the same three probes (code, prose, story, 256 tokens, greedy), and its output SHA was checked against the reference.
| Lever | Mechanism | Baseline t/s | Evidence |
|---|---|---|---|
| Zero-copy state (1 token) | Recurrent state as a view into the cache row instead of a gather; condition: the read row is the write row | 72 → 82 | Byte-identical, Lightning +2% |
| Four rows per wave (K ≤ 512) | The expert down projection left half the lanes idle; 16 instead of 32 lanes per row, reduction with exact zeros | 82 → 86 | Byte-identical, 690/690 tests |
| Flat concat, multi-token down | Concat launched a 256-thread block per channel for five elements; multi-token verification moved to the four-row kernel | Verification −2 ms | df code 132 → 145 |
| Router 257 + passthrough | Shared expert as expert 257 in the GGUF, router extended with its gate row, top-k writes ID and sigmoid weight directly | 86 → 91 | PPL 5.6286 vs 5.6302, 1350 → 1190 kernels |
| Zero-copy for multi-token batches | Same view logic for verification batches; two bugs found along the way (see below) | df +5% | Byte-identical on both target models |
| Q8 epilogue in the gated kernel | silu(gate)·x quantized in the same warp; eligibility determined by consumer shape, not producer shape | 91 → 92 | quantize 120 → 80 per token, byte-identical |
| Lazy drafter injection, gate 200:4 | The shim ran the drafter twice per target decode, even without a draft; buffer features and inject before drafting | Hold step 13.7 → 12.9 ms | df prose 66 → 73 |
| Ordered inter-card copy (staged) | D2H into a pinned host ring on the source stream, event, H2D on the destination stream; faster than the runtime peer path | 92 → 99 | tg32 93 → 102, 4 × 20k stable |
The router with 257 rows
The shared expert cost three small launches per layer, plus sigmoid, multiplication, and addition. Moving it into top-k selection as the ninth expert was the obvious idea, and the first version achieved nothing: extending the ID and weight vectors in the graph cost as much glue as the saved launches gained. Only when the router itself gained a 257th row and the fused top-k kernel wrote ID 256 with its sigmoid weight directly into the output buffers did the glue disappear.
Three traps along the way apply to every later fusion: node order in the ggml graph is a depth-first traversal from outputs, not construction order, so fusion chains must follow actual order. Copy and cast nodes reference themselves, which the legality checker counted as external users and rejected the fusion. And a fusion must not begin at a view whose source lies outside it.
The copy between cards
The operating-point matrix late that evening showed something unexpected: as soon as a model fits on one card, one card is twelve percent faster than two. Switching cards per token cost eight to twelve percent, because since an earlier campaign the inter-device copy had been synchronous through the host. An explicitly ordered two-stage copy through pinned memory with events between stages took tg32 from 93 to 102, beating the runtime’s peer-copy path, which apparently stages inefficiently. This was the final large item: eight percent across the board, because every token and every verification step switches cards once.
Four levers that did not work
- Norm-to-Q8 fusion for expert consumers. Correct, byte-identical, forty fewer launches per token, and exactly zero speed gain: the epilogue cost what the saved launches gained. Retained as cleanup.
- The shared expert as the ninth expert without extending the router. Identical perplexity, neutral speed, for the reason described above. The redesign became the precursor to router 257.
- F16 router. The matvec improved from 13 to 9.5 microseconds, but routing ties broke differently, greedy paths changed, and the prose probe fell five to seven percent. The saved 0.15 milliseconds were below run-to-run variation. Rejected.
- Smaller quantizations. On two cards, Q3_K_S was 29 percent slower than Q4_0 with 8.6 percent worse perplexity; Q2_K was worse still. K-quant kernels are slower on this architecture. Q4_0 stays.
Two bugs nobody was looking for
Extending zero-copy to multi-token batches initially produced divergent output. Bisection found two real bugs. First: graph reuse for hybrid memory checked recurrent fields inline and did not know about the new field; after rollback, a graph with a baked-in view was reused. The second was older and more serious: the registry for prequantized activations found entries by data pointer, but the allocator reuses buffers. Since that morning, the production model Lightning had been feeding unrelated Q8 buffers into experts, MTP acceptance had fallen from 93 to 89 percent, and decode-path perplexity had risen as high as 32 instead of 7.4.
Timeline
- 12:00Start. Baseline 72, df code 135, prose 63, 6k 37. Three hypotheses, no trace in the repository.
- 14:00Kernel trace: 1452 launches per token, 145 GB/s effective. The “bandwidth” hypothesis falls; “pipelining” falls in a pp4/pp8 pretest.
- 16:00Zero-copy state: baseline 82. Drafter-window gate: 6k from 37 to 69.
- 18:00Four rows per wave, flat concat, multi-token down: baseline 86, df code 145.
- 19:30PMIN sweep and acceptance gate: no operating point lifts prose above 71. Finding: prose is draft-poor, not acceptance-poor.
- 20:00Lazy injection, gate 200:4: prose 73. Trunk round 1 (norm-Q8, ninth expert): neutral.
- 20:40Router 257 in four rounds: baseline 91, 1190 kernels per token.
- 21:25Multi-token zero-copy, two bugs found, Lightning restored to reference.
- 22:30Q8 epilogue in the gated kernel: 1150 kernels, 10.25 ms. F16 router measured and rejected.
- 23:30Operating-point matrix: one card beats two; smaller quantizations do not pay off.
- 01:45Staged copy: baseline 99, df code 147, prose 82, 6k 82. End of the lever list.
Assessment
| Operating point | Start (September 2, noon) | End of DX (September 3, 01:45) | After DZ (September 3, evening) |
|---|---|---|---|
| Baseline without drafter | 72 | 99 | 100 |
| DFlash, short / long code | 135 / 118 | 147 / 120 | 145–147 |
| DFlash, prose / story | 63 / 61 | 82 / 80 | 90 / 88 |
| DFlash at 6k context | 37 | 82 | 82 |
| Kernel launches per token / GPU time | 1452 / 13.9 ms | 1150 / 10.2 ms | 1150 / 10.0 ms |
| Lightning (production), code / prose | 95 / 80 | 99–101 / 84–85 | 100 / 84 |
All values: two cards, 256 tokens, temperature 0, first run after loading discarded. Baseline outputs remained byte-identical to reference after every kernel change; df outputs change with reduction order and are validated through acceptance and backend tests.
Epilogue: the final ten percent
By the following evening, the baseline was at 100 and the drafter was below it on prose. The assumption was that the acceptance gate tried too often. Campaign DZ disproved that piece by piece: gate variants neutral, host logits below one percent, feature extraction zero, hook two percent. Only a second kernel trace, this time comparing the DFlash loop against baseline, identified the cause: 235 additional copies per token for a ring of recurrent states needed for drafter rollback. The ring was also shifted on steps without a draft, although it is invalid there by definition. Skipping this brought hold steps to baseline speed and prose to 90.
What remains is not a kernel problem. On prose, the drafter accepts 0.64 drafts per step, and every mechanism given to it converges toward baseline. Exceeding 100 on prose requires a drafter that handles prose, or a baseline whose cost keeps falling: 1150 kernel launches per token still take ten milliseconds, and 40 percent of that is expert matvecs that could be fused.
What the campaign taught us
Trace first, hypothesis second
A kernel trace over 17 tokens corrected three weeks of model calculations. Every campaign now begins with a trace of the current state.
Byte identity as an acceptance criterion
Seven of the eight levers were byte-identical to reference. The only one that was not (router 257) was accepted through perplexity and acceptance checks, with the reason documented.
Neutral is a result
Four levers achieved nothing. They are recorded with measurements and explanations so nobody builds them a second time.
The production model is part of validation
A bug that escaped detection for three campaigns because it affected a different model made Lightning validation mandatory.
One device instead of two, when it fits
Switching cards costs eight to twelve percent per token. For anything that fits on one card, one card is the faster configuration.