480 billion parameters on 32 gigabytes

The model

Flash-Next is a hybrid model: 48 layers, of which 40 are recurrent Gated DeltaNet layers and 8 use attention, plus hyper-connections with four parallel residual streams, a mixture of experts with 512 experts and top-10 routing, and per-layer embeddings (PLE) at layer 1. PLE is a table with 320 million rows, occupying 26.8 gigabytes, 45 percent of the model’s size, addressed per token through n-gram hashes. As an IQ1_S quantization, the model occupies 67.6 gigabytes on disk. The machine has 62 gigabytes of RAM and two cards with 16 each.

So the model fits neither in VRAM nor in RAM. The campaign’s question was not how fast it would run, but whether it would run at all.

First ceiling: a hard drive nobody suspected

In late August, the model ran at 0.2 to 2 tokens per second in the upstream build. The diagnosis was “RAM thrashing”: the model is larger than memory, so pages are constantly swapped. On August 27, the architecture pull request was merged into llama.cpp. During setup, iostat caught our attention: the data drive was 99 percent busy, handling 135 to 142 reads per second of 4 kilobytes each, with a 7.3-millisecond wait and queue depth one. The PLE table was memory-mapped from a hard drive, and the roughly 28 page faults per token were serial head seeks.

After moving the shards to NVMe and placing 30 expert layers symmetrically across both cards, Flash-Next was usable for the first time. But only in the upstream build: the fork did not support the architecture, so it lacked the expert cache, speculation, and all gfx906 kernels.

The port: 614 lines and a sigmoid

Porting the architecture into the fork involved 20 new tensor types, 13 metadata keys, hyper-connections with mixer and combiner, and reuse of the attention, DeltaNet, and MoE components from the Qwen3.5-MoE code. PLE was deliberately omitted at first, with a prominent warning. The model loaded, computed all 48 layers, and produced gibberish, as expected with 45 percent of the model missing.

The PLE port came second: host-side n-gram hashing because ggml supports neither 64-bit integers nor XOR, and dilated causal convolution as an extension of the recurrent state row. This exposed a bug in the reused DeltaNet code: it wrote its convolution state using a compact offset rather than the row stride. As long as the row contained only DeltaNet state, these were identical; with the PLE extension, it would have written into other rows.

The output remained gibberish. The error was not in PLE, but in one line: qwen4exp uses sigmoid in the recurrent layer’s output gate, while Qwen3.5 uses SiLU. Upstream labels this in code as “the one numerical difference.” With the fix, the model completed “The capital of Japan is” with “Tokyo, which is located on the island of Honshu.”

The expert cache

32 of the 48 expert layers must remain on the CPU. Every token pulls its ten experts per layer across PCIe, unless they are already on the card. The LRU expert cache, ported for Lightning in campaign DL, holds recently used experts in slots on the cards and swaps them within a per-step budget.

This put the fork on par with the upstream build with cache, and clearly above DH’s best result. Unlike upstream, it also had a server path with speculation, prompt caching, and profiles.

The head that adds nothing: a refutation in five steps

Flash-Next has a multi-token prediction head that proposes a second token per step. Porting it meant hand-working against a still-open upstream draft: nine consecutive blockers, each a real defect. The most serious was in the batch path. The head reads the wide hyper-connection residual stream, four times 2560 values; without an explicit specification, llama copied only 2560 of them. Three quarters of the head’s input was garbage, and its guesses reflected that: 2.4 percent acceptance on prose. Adding one line that already existed in the neighboring case raised acceptance to 63 percent, and to 79–84 on code.

Yet the head still halved throughput. The search for the cause is the campaign’s most methodologically interesting part, because every obvious explanation was measured and rejected:

• The head needs its KV history. A persistence mode changed nothing. Falsified.
• The head file is incomplete. A dump showed all 34 tensors with correct dimensions. Falsified.
• The head is on the hard drive. Copied to NVMe: plus 0.5 t/s. Falsified.
• The head displaces VRAM. It occupies 1.9 GiB on card 1 and displaces 0.66 GiB of cache. That does not explain a factor of 3.3. Falsified.
• Chained drafts will pay off. With three drafts, acceptance fell to 27–31 percent. Falsified; one draft is the operating point.

What remained was speculation’s underlying assumption itself: verifying two tokens costs roughly as much as decoding one. The step breakdown in campaign DT quantified it: the head costs 2.2 milliseconds per step, two percent of step time; two-token verification costs 1.9 single-token decodes. The batch amortizes nothing, because the offloaded layers’ experts execute on the CPU, and two tokens pull two largely different expert sets. At 24k context, things did not improve: 13.0 versus 6.4–7.4 t/s, on German technical prose with 26–43 percent acceptance.

The 14 milliseconds that were 4.5

A gap separated graph computation from measured step time, estimated at 14 milliseconds in campaign DU. A section timer in the decode path showed that estimate was wrong: the split sum had been measured with a synchronization per split, and the verification time came from a different run with ten milliseconds of variation. The real gap was 4.5 milliseconds. Two items accounted for it.

The smaller one: expert-cache table updates, up to 128 synchronous four-byte copies per step. Batching them into one copy per changed layer saved 0.25 milliseconds. The larger one: the graph was rebuilt and allocated on every decode, 1249 nodes across three backends, costing 3.5 milliseconds, because the PLE input class categorically prohibited reuse. It held its n-gram history in the graph object and assumed a reused graph would lose it.

With reuse: plus 7.7 percent. Then outputs diverged between reuse on and off, and perplexity became the referee: bit-identical on 512-token chunks, yet still no acquittal, because both paths were wrong in different ways. Without reuse, the trunk lost history on every token; with reuse, on every shape change. Since the port, every Flash-Next decode token had been computed without its n-gram context.

This was a quality finding, not a speed finding. It retrospectively explained why the MTP head achieved only 57 percent acceptance rather than the upstream-reported 70–80: it predicted a model with context, but verification used one without. After the port, both paths were bit-identical, reuse was stateless and enabled by default, and the trunk reached 22.0 t/s.

Timeline

• 27.08.DH: Hard drive exposed; NVMe, 30 GPU layers. 0.2 → 13.4 t/s, upstream build only.
• 30./31.08.DO: Architecture port into the fork, 614 graph lines, PLE omitted, incoherent output.
• 31.08.DR: PLE port, a DeltaNet offset bug found, the sigmoid: Flash-Next computes correctly in the fork. LRU-128: 18 t/s.
• 31.08.–01.09.DS: MTP head ported, nine blockers, acceptance 2 → 63%. The head halves throughput; five explanations falsified.
• 01.09.DT: Step breakdown: verification costs 1.9 decodes, the head 2%. Long context does not reverse the sign. Blocker 11 emerges.
• 02.09.DV: Resident PLE instead of mmap, graph reuse, batched tables, the lost n-gram history: 22.0 t/s with 3% lower PPL.
• 02./03.09.DW, DY: The expert-cache race, mitigated; a separate case study.

Assessment

Flash-Next runs in the fork at 22 tokens per second for one user at 4k context, 13 at 24k, with correct output and perplexity on par with prefill. The MTP head is ported, accurate, and disabled because it adds no value on this system. Remaining options: more cache slots when VRAM is free, a two-token variant of the tuned expert kernels for verification, and an index rather than a scan for predecessor lookup beyond 32k context.

What the campaign taught us