A crash in three out of nine runs

The symptom

Qwen3.8-Flash-Next is too large for 32 GiB of VRAM. The fork runs 32 of its expert layers on the CPU and keeps recently used experts in an LRU cache with 128 slots on the cards. This lets the model run at 22 tokens per second. In a test matrix with 24k prompts at 32k context, the server died on the second identical request with ROCm error: an illegal memory access. It was stable without cache, with cache threshold 1, without checkpoints, and under full serialization of kernels and copies.

This produced the problem’s first name: “race in the multi-token cache path after a checkpoint hit.” Three nouns, three premises. By the end of the first campaign, none remained.

Day one: DW, or how three premises fall

The bisection that narrows nothing down

The code offered three suspects: decode returns while the graph is still running on the cards; the expert cache writes its tables through a per-thread stream without ordering against compute streams; and checkpoint hits copy KV buffers between streams. None alone explained an illegal access, because corrupted weights produce wrong values, not wrong addresses.

A switch synchronizing after every graph node made the crash disappear but identified nothing, because synchronization closes the window. Bisection by operation type showed that syncs after GET_ROWS, copies, flash attention, or MUL_MAT held, while syncs after MUL_MAT_ID or recurrent kernels did not. Four of six filters close the window, too many to identify a culprit. The question was position, not type.

First premise: there was never a checkpoint hit

Checkpoints were not enabled in any test script, and identical prompts do not produce one in the anchor path anyway. The “without checkpoints” configuration was identical to the other one. What remained: a second long request on a context with eight KV streams, crashing in seven out of nine unmasked runs.

Second premise: the multi-token path was never involved

The request flow contains no batch with two to four tokens: prompts have eight, prefill chunks 512, and decode steps one. The threshold could not change the graph. When threshold 1 also crashed twice, it became clear that the earlier “stable” run at threshold 1 had been luck. The split logs also showed exactly where: in the second request’s prefill chunk, in a subgraph with 1249 nodes where the scheduler uploads the used experts’ CPU weights to the card.

Third premise: “stable” means nothing

When even the crash configuration survived three requests, statistics corrected the method. Across all unmasked runs: nine crashes, four survivors, always on request two, never on three. A race with roughly 70 percent incidence, whose window opens only when switching to the second sequence. A crash is conclusive; a single survival offers only 70 percent confidence. Acquittal requires four repetitions to push chance below one percent.

Tools that make the problem disappear

Without root, there was no kernel log with the faulting page address, no perf, no gdb, and no compute sanitizer for gfx906. That left runtime tracing. Under AMD_LOG_LEVEL=3, the ROCm 7 HIP runtime wrote nothing, but the server survived twice. Under rocprofv3, the crash reproduced, but the abort path wrote no trace; after changing to an exit path, the server survived again, twice. Both tools alter dispatch timing and close the window. Tracing was a dead end. Behavioral experiments remained.

The mitigation and the first cause

Position bisection suggested a mitigation without an identified culprit: a sync after every subgraph with at least 256 nodes, meaning only the large prefill splits. Four out of four stable, prefill within noise, decode untouched. Then came two hypotheses about the cause. Asynchronous pageable host-to-device copies were ruled out by code inspection. The other held: intermediate state moves between cards for every chunk through a peer copy on the source stream, without peer access ever having been enabled, because two Pro VII cards over PCIe do not support it. Without P2P, the runtime performs the copy in two stages through host memory, while events order only the first stage. A synchronous fallback for cross-device copies: six out of six stable. It became the default, and blanket split synchronization returned to opt-in.

Day one ended with a named culprit, a fix with no measurable cost, and lingering unease: why a delayed copy would cause an illegal access rather than merely a wrong value remained conjecture.

Day two: DY, or how the culprit is cleared

It was not the copy

The next day, campaign DX for the 35B model introduced an explicitly ordered two-stage copy between cards: device-to-host into a pinned ring on the source stream, event, host-to-device on the destination stream, event back. It was faster than the runtime path and gained eight percent on the 35B. In the Flash-Next crash configuration, it crashed in four out of six runs. The DW attribution therefore had to be corrected: the synchronous fallback worked because it completely synchronized both cards for every copy, not because copy ordering was the cause.

So back to the beginning, this time with six-run series and a baseline rate. First new hypothesis: ordering between graph and cache step. A sync before the cache step: four out of six. Rejected. Second: cache operations inside the graph itself. With cache allocated but its graph operations disabled: five out of six. Part of the window, not all of it.

A real bug that was not the culprit

Reading the cache code revealed something: it maps all uncached experts for a token to the same dummy slot, so a token can contain several identical expert IDs. The helper kernel that sorts IDs by expert for MoE prefill stores at most one hit per token but counts every ID. The row lists developed gaps containing uninitialized indices. A per-token bitmask fixed this.

The first version of the fix had a prefix-scan bug: it shuffled the constant through the lanes instead of the running sum. The result was not a crash but a hang: the HSA queue reported a memory fault, one CPU thread spun at a hundred percent, both cards stood idle, and the server’s health check kept responding. The test scripts had reported “stable” because they checked only health. A dump of kernel outputs for a five-token case (bounds 0 2 4 4 4 instead of 0 2 4 4 5) exposed the scan error. Once corrected, the kernel passed all 690 backend tests, perplexity stayed at 3.3553 across all variants, and the test scripts now classify a hang as a crash.

This fixed a real correctness bug. With it: four out of six stable. Not the culprit.

Narrowing it down through serialization

Two series provided the key insight. With complete serialization of all kernel launches, there was no crash; with serialization of copies alone or kernels alone, each had one. So this was a race between a kernel and a copy on different streams, not a deterministic out-of-bounds access. An ID guard, a check kernel before every MUL_MAT_ID and GET_ROWS that writes out-of-range IDs to a device flag read by the host only at graph end, never triggered in six runs despite barely changing timing. The expert IDs were never wrong. Those six runs were nevertheless stable, because of the one sync per split that the guard needed to read its flag.

The final two series narrowed this down. A sync per split only in large graphs, meaning only prefill: four out of six. A sync per split in every graph, including decode graphs and the tiny two- and twenty-one-node graphs between requests: six out of six, with no measurable cost in decode or prefill.

And production?

The mitigation entered the fork as a switch. A follow-up comparison showed that staged copying adds nothing to the Flash-Next profile: 19–21 tokens per second in both variants, with identical outputs. The production profile retained DW’s synchronous fallback, which already holds six out of six there. The 35B profiles keep the fast staged copy; they have neither cache nor CPU experts, and around twenty matrix runs there produced no crash.

Assessment

The culprit remains unidentified. Remaining candidates are host copies between requests (checkpoint and state copies) racing with still-running streams, and the cache step through the per-thread stream, which series T6 excludes only as the main cause. Both can be targeted with the ID guard, an afternoon’s work. The campaign nevertheless leaves more than a mitigation: a method using baseline rates, a set of tools that remain in the fork, and certainty that the cross-device copy blamed on day one was not the cause.

What the campaign taught us