Chasing 50/60: Two silent bugs and an overturned verdict
One evening, one goal: 50 tokens/s in a single stream, 60 aggregate, on a 150-watt MI50 with a 27-billion-parameter hybrid model. The 60 target fell. But not because of the new kernel built for it: it fell because of two existing bugs that had been present in every measurement for days, unseen because every verification run was exactly one request long. And the solo record came from an idea buried eight days earlier with a sound calculation.
the 60 target, formally confirmed
91.6% of the 50 target
together ≈ ⅔ of the gain
none merely asserted
A target and an apparently exhausted inventory
The assignment was a number: 50 tokens/s single-stream, 60 aggregate, on Q4_0. Before this work: roughly 41–42 solo, 57 aggregate. The machine: a single MI50 with 16 GB and a hard 150-watt cap. The model: a DeltaNet hybrid whose multi-token prediction head (MTP) was already running depth-1 speculation. Each step verifies the guessed token alongside the normal one for free, with acceptance above 90% on code.
The obvious levers had been exhausted. What followed was fifteen hypotheses in one night, and the realization that the biggest gains were not in new kernels, but in existing code that was silently malfunctioning.
The head moves into the main graph, and the reference wins Falsified
The first major hypothesis: the MTP head lives in a second context, so every step pays for a second GPU invocation (a “flush,” approximately 2.3 ms). Moving the head into the main graph as a tail would eliminate the flush entirely, generating the draft in the same invocation as verification.
The redesign was fully implemented: shifted semantics for the head inputs, a 4-byte draft instead of a megabyte-sized logits row, and free rollback through the existing checkpoint. A purpose-built comparison instrument confirmed correctness: 138 out of 144 drafts identical to the second-context path (the rest were numerical ties). Yet the redesign lost the race: 41.7 versus 44.6.
Why a correct redesign can lose: After each rejected draft, the in-graph head predicts from the dead branch, so the following step runs without a draft. The old second context, by contrast, alwayspredicts. This one structural difference (E[tokens/step] = 2/(2−α) instead of 1+α) costs more than the saved flush gains. The redesign stayed in the tree as an option; the stopwatch delivered the verdict.
Two bugs in every measurement, and a silent log
While measuring the redesign, the server crashed, but only from the second request onward. Every previous byte-identity verification had been exactly one request long: green for weeks, and blind to anything that breaks later in the lifecycle. The crash log showed only a bare abort line; our own log filter, built to keep benchmark output clean, also swallowed the error details.
Make errors visible. The log filter now always lets error lines through. The cause appeared immediately: “memory range is already mapped”: host-memory pinning without a matching release. Every request registered buffers, none released them; the allocator recycled the blocks, and the next request tripped over their remains.
Take the anomaly seriously.
In the same error lines: “invalid backend sampled token … batch.logits[0]!= true”. The draft-read path queried row 0 instead of the last row. After every accepted draft (two entries in the head batch), the read silently failed and returned “no draft.” The signature had been in every statistic for days: drafted = steps −
accepted. It was visible. It was simply never questioned.
Both fixes, one jump. Solo: 41–42 → 44.6. And the aggregate, whose transfer path had also been affected by the pinning leak, later jumped from 57.2 across the finish line.
The real lesson: both bugs were observabilityfailures. A verification protocol that never ran two requests; a log filter that filtered out errors; an anomaly in our own figures that nobody read. The most expensive code was already written. What was missing was attention.
Depth-2 speculation: buried eight days ago, a record today Confirmed
Eight days earlier, “depth 2,” two guessed tokens per step instead of one, had been rejected with a sound calculation: the ceiling rose by 5.5%, while the second head pass cost 5%. A wash; do not build. The calculation was correct. For the world as it was then: 74-ms steps, 3.9-ms flushes, α₁ = 0.91.
The world had changed: 43.7-ms steps, 2.3-ms flushes, α₁ = 0.93, and an existing probe that measured the decisive quantity before any implementation: how often does the second, chained token hit? On Q4_0, the answer was: 74.2% on code, 41.4% on prose. For code, this overturned the old calculation.
| Workload | Depth 1 | Depth 2 | Verdict |
|---|---|---|---|
| Code | 44.6 t/s | 45.80 t/s | +2.6%, new record |
| Prose | 37.4 t/s | 34.7 t/s | −7.7%, retain depth 1 |
Notably, the old cost calculation was confirmed by measurement: each additional verification row actually costs 25% of step time (predicted: 33%). On prose, where the second token hits only 41% of the time, that row consumes the gain. Depth 2 is therefore a switch for code workloads, not a default.
Keeping track of candidates pays off: a candidate that lost under old costs won under new ones. The earlier verdict was not a mistake; it was tied to its cost basis. Filing buried ideas with explicit revival conditions lets you harvest them in two hours instead of two days.
A probe finds no experts, because there are none
For the remaining gap, there was an attractive theory: the model was a mixture of experts, each verification row loaded new expert weights, and a multi-token kernel could share the weight stream. A probe was built to count expert overlap between neighboring tokens, and found: nothing. Wired into four dispatch points, zero hits.
The third wiring attempt proved why: the 27B has no expert nodes on its compute path. It is dense. The “128 experts” in our project notes belonged to the sibling model (35B-A3B), a transcription error that had persisted for weeks and suggested an entire optimization direction that did not exist. The additional-row costs instead come from the linear DeltaNet scan across 48 GDN layers: a physical cost that weight sharing cannot optimize away.
Quoting your own memory is not measurement. The expert assumption was in the notes and had never been checked against the graph. It took a probe to bring it down. The probe remains in the tree: it is immediately useful for the actual MoE model (35B-A3B).
Everything into the graph, and physics answers
Two final redesigns executed draft 1 and draft 2 entirely in-graph: zero head invocations per step, with hand-built attention against the KV cache plus a concatenated self column, a new position input, and F16 support in the concat kernel. Both ran stably and byte-reproducibly; one even produced exactly the same drafts, steps, and text as the host path, a clean correctness proof for the whole chain.
They did not get faster: 44.6 and 45.82 versus 45.80. The second head invocation’s 2 ms turned out to be real GPU work in the head block, not invocation overhead. Moving work into the graph does not eliminate it. The inventory was now not merely mapped, but measured to exhaustion: flush structure, fusions, draft depth, parameter sweeps, all built, all timed.
| Hypothesis | Built | Verdict |
|---|---|---|
| Head in main graph (TAIL) | ✓ + byte-verified | Loses to the redo penalty |
| Depth-2 solo (host chain) | ✓ | +2.6%, record 45.80 |
| Depth 2 in aggregate | ✓ | Negative (more traffic on a busy GPU) |
| Fusions across 3–4 columns | ✓ + byte-identical | Neutral (heads too small) |
| Expert-overlap kernel | Probe ✓ | Not applicable: model is dense |
| d1+d2 in-graph (2 variants) | ✓ + correctness proof | 44.6 / 45.82, no gain |
| k-cap and thread-pool sweeps | ✓ | Negative / neutral |
| Two pre-existing bug fixes | ✓ | ≈ ⅔ of the total gain |
60 reached, 50 characterized, plus a sawtooth
The formal measurement still had one footnote waiting: initial runs oscillated between 17 and 26 t/s, not because of the code, but because of a forgotten system state. A DVFS experiment the previous day had pinned the performance level to high; at the 150-watt cap, the highest clock state cannot be sustained, so the power governor oscillates. After resetting it, cold, with a cooling gate:
| Target | Measured | Status |
|---|---|---|
| 60 t/s aggregate (3 streams) | 60.67 / 60.43: both rounds above 60 | Reached |
| 50 t/s solo | 45.80 (code, depth 2): 91.6% of target | Open |
The remaining 8.4% is no longer a wish list but a characterization: the solo step draws only approximately 400 of the approximately 580 GB/s actually achievable. The ceiling is not bandwidth, but the fixed-cost structure of the linear GDN scan across 48 layers, plus approximately 2 ms of real head work per draft stage. The next jump requires new kernel territory: scan fusion across layers and wave profiling of multi-column GEMVs. That is a new program. This one ends with a reached target, an honest number, and an empty but fully measured inventory.
Five lessons from one night
Single-request verifications are blind. Lifecycle bugs (leaks, duplicate registrations) require regression tests with at least two requests per server lifetime.
Log filters must not filter out errors. A crash log without a cause line is self-inflicted blindness.
Question anomalies in your own figures.
drafted = steps − accepted appeared in every output for days; the equation was the bug report.
Verdicts age with their cost basis. An eight-day-old “do not build” became a record under new costs. Buried ideas need revival conditions.
Notes are not measurements. The “128 experts” from project memory survived for weeks, until a probe interrogated the graph itself.