Inference correctness · Engineering note
When a passing test misses the state bug.
A batch-512 perplexity check did not expose missing token history. Changing the test to batch size 1 did.
This investigation came out of my gfx906 inference project, while working on the qwen4exp / Flash-Next port. It is a development finding from that implementation, not a claim about every deployment of the model.
A performance question became a correctness question
The runtime was rebuilding a graph during decoding. Reusing it reduced overhead, but the generated output changed. That made performance an insufficient acceptance criterion: first, the difference needed an explanation.
The port's per-layer embedding path uses previous tokens to construct n-gram inputs. That history was tied to the graph object. Rebuilding the graph discarded it; graph reuse could still lose it when the graph shape changed. The two execution paths were therefore not an independent correctness reference for each other.
What the original test did not exercise
The batch-512 perplexity comparison did not reveal the problem. It failed to cover the repeated history transitions of token-by-token decoding. A batch-size-1 evaluation made that distinction visible.
| Execution path | Perplexity |
|---|---|
| Batch 512 reference | 2.4048 |
| Batch 1, previous history handling | 2.4781 |
| Batch 1, KV-cell history port | 2.3978 |
The old batch-1 value was about 3% higher than the batch-512 reference. The post-fix value is evidence for this test case, not a claim of a 3% improvement in general model quality. The runs have not been independently replicated.
Move history to the state that survives
The fix followed the upstream pattern: retain tokens in KV-cache cells and resolve predecessors by their positions, rather than making a graph object responsible for persistent token history. The documented checks covered microbatch boundaries, graph reuse, and sequence copies.
A separate host-state restore path was not fully covered by that change. It needs its own regression test. Passing the sequence-copy path does not establish correctness of every serialization and restore path.
The testing lesson
- Test transitions, not only steady state. Include single-token decode, batch-shape changes, and graph rebuilds.
- Use a reference with a different failure mode. Two paths agreeing can mean they share a bug.
- Keep performance and correctness separate. Lower latency does not explain a changed answer.
- Make the remaining gap explicit. Cache copying, rollback, and host restore are distinct tests.
Evidence and status
Source: development log DV-profil-und-14ms-02-09-2026.md, campaign DV, and the KV-cell history change recorded as 3c63b0f. These are references within the private project archive, not public download links. A standalone reproduction package is not yet available.
This is an engineering note, not a peer-reviewed paper. The implementation follows existing upstream state-management work; the contribution described here is the integration investigation, the test blind spot, and its correction.