Six hypotheses, four refutations, +31% throughput
An AMD Instinct MI50 from 2018 used only 31% of its memory bandwidth when generating text. The obvious explanation was wrong. So was the second. How a team of AI agents found the cause, and what the project’s most expensive failure ultimately delivered.
232.1 → 320.0 tokens/s
88.6 → 118.1 tokens/s
after fusion and epilogues
including our own leading hypothesis
The problem: a card waiting for itself
The MI50 is a data-center card with 16 GB of HBM2 and roughly 1 TB/s of theoretical memory bandwidth. During text generation, the decode step that produces one token after another, language-model inference is almost pure memory work: every model weight must pass through the chip once for every token. Across all model sizes, the achieved rate was consistently around 320 GB/s, or 31% of the possible bandwidth.
A decode step consisted of 545 individual GPU kernel launches: normalization, quantization, projections, positional encoding, attention, feed-forward, repeated 28 times, plus the output layer. This led to the project’s central hypothesis, and it sounded compelling.
Hypothesis ledger
Every hypothesis was implemented, measured, and either confirmed or rejected. The verdicts below are measurements, not opinions; each includes the number that decided it.
Decode is dispatch-limited: 545 kernel launches per token never let the card get going.
Timestamp analysis of every kernel transition: gaps between kernels account for 1–2% of token time. An enqueue costs the CPU 1.86 µs, while the average kernel runs longer, so the queue stays filled. The losses are inside the kernels, not between them.
A persistent megakernel, one launch per token for all 28 layers, unlocks threefold potential.
Fully implemented: six internal phases, attention with context splitting, synchronization through a custom grid barrier. First measurement 234.6 t/s versus 269.4 for the comparison build: −13%. After porting the memory technique discovered later: +5.3%. Once the same gain reached the normal path: +2.8%. On the 7B model, however, +13.5%; there the kernel reaches 75% of the bandwidth reference instead of 34%. The threefold potential does not materialize, but the hypothesis is correct depending on model size.
The bandwidth gap comes from the q4_0 format: 18-byte blocks cannot be loaded with proper alignment.
The prime suspect was innocent. Machine code showed five memory instructions per loop iteration, three for the activation, which is already cached and generates zero main-memory traffic, but occupies 60% of memory instructions. The real cause.
More concurrently running thread groups hide memory latency better.
First, the prototype showed that 240 instead of 168 resident workgroups cost 4.3%. Then the stronger finding: a kernel reaching the hardware maximum of 40 waves per compute unit collapses to 192 GB/s, while just one fewer wave (through 1664 bytes of unused LDS at launch) delivers 441 GB/s, a factor of 2.3, with byte-identical kernel code. Full occupancy is a cliff on this architecture, not a goal.
Fewer, longer kernels help by fusing neighboring compute steps.
Q, K, and V projections in one launch (−112), normalization writes the quantized format directly (−56), positional encoding and cache writes as epilogues (−28 each). Result: +16% on the 1.5B, with bit-exactly unchanged results.
Standard remedies work: HIP graphs against launch costs, draft speculation against latency.
HIP graphs are slower on this architecture and leak memory (~370 B per node and invocation, demonstrated with a standalone reproducer). Draft speculation using a 1B model for an 8B model: +0.9%. Both rejected.
What the failure left behind
The failed megakernel delivered something nobody had before: a reference for what this card can actually achieve. Its output head, a trivially parallel matrix-vector kernel on the same chip, reached 768 GB/s. Not a data-sheet figure, but a measurement.
For the first time, this quantified that the production kernels used only 55% of that bandwidth. A ladder of isolated microbenchmarks then cleanly separated the hypotheses:
The second row is the answer: +47% simply by omitting activation loads. The third shows that the compute chain costs nothing; the fourth independently confirms the reference. The resulting redesign, a full data block per lane through wide loads, multiple output rows per workgroup, delivered +20.5% on the 7B model, reproducibly confirmed by an A/B/A run to within 0.04 t/s.
Benchmarks
All figures come from alternating measurement arms with 150-second cooldowns (explained below), five repetitions each, flash attention enabled.
| Model | Initial | After fusion | After memory fix | Current | Total |
|---|---|---|---|---|---|
| Qwen2.5-1.5B Q4_0 | 232.1 | 269.4 | 297.5 | 320.0 | +38% |
| Qwen2.5-7B Q4_0 | 88.6 | 93.0 | 114.5 | 118.1 | +33% |
| Llama-2-7B Q4_0 | 96.8¹ | 117.4 | +21% | ||
| Llama-3.1-8B Q4_K_M | 70.6 | 70.8 | 73.7 | 73.9 | +5% |
¹ The Llama-2 gain has an uncomfortable history: much of it fixes a self-inflicted regression, a kernel-selection window validated on only one model that slowed the most common matrix width in the field (K=4096: Llama-2, Mistral, Llama-3) by 16%, while gaining 0.4% on the validated case. Details below under “Near-miss conclusions.”
| Metric | Before | After | Note |
|---|---|---|---|
| Kernel launches / token (1.5B) | 545 | 294 | Fusion + epilogues |
| Effective bandwidth (7B) | 384 GB/s | 471 GB/s | Out of 768 achievable |
| Grid barrier, custom implementation | 10.8–13 µs | 1.07 µs | Versus AMD Cooperative Groups |
| Kernel execution floor | 1.3–2.1 µs | 3.2–4 µs | Isolated versus in the decode stream |
| Occupancy cliff (single matvec) | 192 GB/s | 441 GB/s | One fewer wave per CU |
| Wait instructions in the matvec hot loop | 13–15 | 7–9 | After address-space correction |
| LDS instructions per weight byte | 0.556 | 0.056 | Data-layout redesign |
Why the megakernel won, and why that does not help
Breaking down its advantage explains the recommendation against building it. Positive items are gains over the normal path; the negative item is the cost of persistence itself:
The persistence-specific balance is negative: barriers cost 333 µs and save only 129 µs in launch gaps, a net −204 µs. The kernel wins despite its persistence, not because of it. Three of its four gains do not depend on persistence at all and were subsequently ported into the normal path. Its lead then shrank from +5.3% to +2.8%.
The size effect: the same method, four times the gain
Porting to the 7B model was the last open question, and it changed the assessment. The estimate had predicted +2 to +5%, assuming the advantage came from matrix-vector multiplication. It does not; the normal path is actually faster there.
| Model | Normal path | Persistent | Δ | Long context | Bandwidth achieved |
|---|---|---|---|---|---|
| Qwen2.5-1.5B Q4_0 | 320.0 | 317.5 | −0.8% | Advantage disappears | 34% of reference |
| Qwen2.5-7B Q4_0 | 118.1 | 130.4 | +10.4% | +11.4% | 75% of reference |
This table shows the final state after all portable parts of the prototype’s advantage were integrated into the normal path, most recently positional encoding as an epilogue to the preceding step (+4.0% on 1.5B, +2.3% on 7B). On the small model, the normal path has now overtaken the megakernel. On the large one, it remains 10.4% behind, and that gap no longer melts away, because every grid-wide dependency in the normal path costs a kernel boundary of 3.2–4 µs, where the persistent kernel pays 1.15 µs. That is precisely persistence, and it cannot be fused away.
The explanation is in the last column. The large model has enough work between synchronization points to reach streaming behavior at all. Its advantage comes from 1302 µs per token spent in normal-path kernels that perform no computation: 708 µs of normalization and quantization across 143 launches at the execution floor, 296 µs of positional encoding, 136 µs of attention combination, and 162 µs of launch gaps. Exactly the costs a persistent kernel does not have by construction.
How the agents searched
The investigation used specialized AI agents, each handling one stage: forensics, fusion, prototype, root-cause analysis, and porting. The decisive factor was not their compute power, but the instruments used to separate claims from measurements. Twelve techniques that proved useful:
Kernel inventory through a profiler trace
Every kernel invocation in a decode step captured with name, duration, and timestamp. The problem: where does a token end? Solved by using the output layer as a marker; it runs exactly once per token and cleanly segments the stream.
Found: 545 launches/token, distributed across 19 kernels per layerGap analysis instead of summing durations
Not adding kernel runtimes, but measuring the differences between one kernel’s end and the next start. This metric alone falsified the project’s leading hypothesis.
Found: gaps only 1–2%; the cause is inside kernelsIsolated microprograms
Launch, synchronization, and transfer costs measured in standalone miniature programs rather than estimated from the overall picture, then cross-checked against trace analysis.
Found: enqueue 1.86 µs, barrier 1.07 µs, transfer 49.6 µsReading machine code
The decisive finding came not from a profiler, but from disassembly of the generated kernel: count memory instructions, check their width, inspect wait placement.
Found: 3 of 5 memory instructions serve cached dataDifferential benchmark ladder
Rather than optimizing a guess, variants were built that each omit one ingredient. The difference between two rungs identifies the cause without having to guess it in advance.
Found: +47% when activation loads are omittedSwitchable changes
Every optimization received a runtime switch. This allows both states to be compared in the same program: no rebuild, no version differences, no excuses.
Enabled: clean A/B comparisons across 38 commitsA/B/A instead of A/B
After every comparison, the initial state was measured again. If it failed to return to its own value, the measurement was worthless.
Confirmed: 95.04 → 114.54 → 95.08 t/s, returning within 0.04A second reference to classify errors
When the prototype exceeded the accuracy threshold, the threshold was not raised. Instead, a second reference implementation was created with identical code but single rather than double precision.
Proven: the deviation is a rounding artifact, not a computation errorLive bit comparison during operation
Verification modes compare every generated buffer byte by byte against the reference implementation during real inference, across thousands of buffers rather than a handful of test cases.
Demonstrated: over 3000 buffers, zero deviationsDeliberately provoking failure
Before building a persistent kernel, the condition that makes it hang was forced: more thread groups than can fit on the chip simultaneously.
Consequence: a runtime check became mandatory, not optionalCross-check through reversal
To prove a change does not alter results, it was reverted and checked for exactly the same numbers and text output.
Demonstrated: bit-identical to the initial stateMeasure alternatives, do not assert them
To answer “would splitting outperform the monolith?”, the kernel was restructured so individual branches could be compiled and measured separately instead of estimating the answer.
Result: splitting only approximately 10% better, at best parityFour traps that would have fabricated success
Each of these traps would have introduced a false result into the assessment. All four were caught by verification mechanisms, three only after the mechanism itself had been refined.
1504 GB/s: one and a half times the physically possible
The analysis hard-coded the bytes of a full layer while the prototype executed only 7% of it. The same line extrapolated 999 tokens/s. Every bandwidth figure has since been interpreted against a concurrent reference.
A threshold that could conveniently have been raised
The prototype exceeded the error threshold by 78%. Instead of relaxing it, the yardstick was changed: a single-precision reference showed the same deviation, and at half the positions an even larger one.
An address space the compiler could not infer
Because the pointer type was ambiguous, the compiler emitted a generic rather than specialized load instruction. It counts in two wait queues simultaneously, forcing every following access to serialize individually.
A verification mode checks only what it checks, four times
Four bit-level traps, and each time the existing verification reported zero errors because it compared the wrong stage: downstream quantization instead of raw values, the f32 intermediate instead of the f16 final values, and the fourth time exactly the reverse. The fix for the third trap (conversion directly at the expression, one rounding) would have been precisely wrong for the sibling variant, where the double-rounded value matched the reference. There is no universally correct verification stage, only the stage where the change acts.
A gate validated on one model is not validated
A kernel-selection window was measured on one model (K=3584) and applied to the whole range through K=4096. Result: 16% loss at the field’s most common matrix width, 0.4% gain on the validated case. The error persisted unnoticed for weeks and surfaced only because another measurement happened to hit the right model. The rule since then: thresholds are measured on at least three architectures with different widths.
The hardware maximum is a cliff
The project’s slowest kernel was also its “most efficient”: 23 registers, no LDS, maximum occupancy of 40 waves per compute unit, and at that point 2.3 times slower than necessary with identical cache counters. The fingerprint of this cliff: kernel time is independent of data volume. The only effective diagnostic control: unused dynamic LDS at launch, reducing occupancy by one wave without changing a byte of code.
And a measurement error that would have invalidated every series
The card is capped at 150 watts and throttles to half speed under uninterrupted load: the 7B model falls from 92.7 to 44.9 tokens/s. Running a measurement series without pauses means that from the third row onward, you measure temperature rather than code. Every number in this case study comes from alternating arms with 150-second cooldowns.
What correctness was measured against
Speed without correctness evidence is worthless in this field: a kernel that computes incorrectly can be arbitrarily fast. The same anchors applied after every single step:
| Check | Criterion | Result |
|---|---|---|
| Perplexity, Qwen2.5-1.5B | Bit-exactly unchanged | 9.4145 |
| Perplexity, Llama-3.1-8B | Bit-exactly unchanged | 7.0867 |
| Matrix multiplication | All green | 1066 / 1066 |
| Flash attention | All green | 3344 / 3344 |
| Expert routing | All green | 690 / 690 |
| Prototype endurance run | No deviation | 10.000 iterations |
| Token by token against reference | Identical selection | 64 / 65 |
| Text comparison, 300 tokens | Byte-identical | Passed |
The one deviation in the token-by-token comparison occurred exactly at the position with the smallest margin between the best and second-best choices. There the last bit decides, and the prototype deliberately uses a different format for the output layer.
What a carefully measured failure is worth
The criterion was fixed in advance: harvest the portable parts of the prototype’s advantage into the normal path, and if a gap of five percent or more remains, integration is justified. The harvest is complete, and the answer is opposite for the two model sizes.
For small models, the megakernel is finished: the normal path has overtaken it at 320.0 versus 317.5 tokens/s. Reducing 545 kernel launches to 266 achieved exactly what the monolith was meant to achieve, without its costs. For the 7B class, the normal path remains 10.4% behind, justifying integration for the first time, with open caveats: fixed model geometry, only one quantization format, and the prototype figure is an extrapolation that integration would first have to demonstrate end to end.
The real return lies elsewhere. Without the megakernel being built, there would be no 768 GB/s reference; without that reference, no diagnosis of activation loads; without dissecting its victory, no quantization fix. The +31% comes almost entirely from insights produced by a failure.
Roughly half of the original bandwidth gap is closed, but not where the hypothesis placed it. It was never in the number of kernels, but in their memory-access efficiency: wasted load instructions for already cached data, an address space the compiler could not infer, and dozens of kernel launches for microseconds of work.
What transfers beyond this card
Ordered by transferability, not effect size:
Methodology (applies everywhere): The differential ladder that identifies a cause instead of guessing it. A/B/A rather than A/B: returning to the initial value authenticates the measurement. The insight that verification checks only the stage it targets. And the distinction between ranking and recommendation: a top result measures distance from the status quo; a build decision weighs it against downstream costs.
Architecture knowledge (applies to GCN, partly beyond): Full occupancy as a cliff, with its fingerprint and diagnostic control. The 3–4 µs kernel execution floor in a live stream: every kernel with less than ten microseconds of compute is a fusion candidate. The finding that load instructions for cached data cost throughput even when they generate no memory traffic.
Project result (applies to this card): The throughput gains themselves, the repaired formats, the ranked lever list with a measured value behind every entry, including the eight rejected entries. A documented dead end is a result: it saves the next person the journey.
A carefully measured negative result is not a lost project. It is a measurement instrument.