Built-in speculation: When the model is its own predictor
Classic draft speculation delivered +0.9% on this card: a failure. The same mechanism using the model’s built-in multi-token prediction head: +14.5% on code, with 97.7% acceptance. The difference is not the algorithm, but who trained the predictor. A two-minute measurement before implementation predicted the result exactly.
almost every other step is free
93.4 → 106.9 tokens/s
instead of a separate model
before implementation
A failure with a known cause
Speculative decoding is the standard recipe for overcoming the slowness of token-by-token generation: a small predictor guesses the next token, and the large model verifies the proposal in the same pass. If the guess is right, you get two tokens for the price of one. On this card, the recipe had failed: a 1B model predicting for an 8B model delivered a measured +0.9%. The draft model’s full forward pass consumed the gain, and the unrelated model’s acceptance rate was mediocre.
Then Nemotron 3.5 Lightning joined the roster with its predictor built in: a multi-token prediction block (MTP), a single additional layer trained jointly with the model by its manufacturer. It uses an intermediate state already computed to predict the token after next. Draft cost: one layer out of 52 instead of an entire second model. And because the head was trained with the model, it knows its quirks.
The method’s safety guarantee: with deterministic selection, the output is equivalent to normal decoding by construction. A miswired predictor does not produce incorrect text; it only lowers the acceptance rate. Correctness and speed are clearly separate questions.
Calibrate before you build
The most expensive question in any speculative approach is: what does it cost to verify two tokens instead of one? In memory-bound decoding, a batch of two should be almost free: the model weights are read once anyway. Should be. But this model does not fit entirely in GPU memory; some expert weights reside in system RAM. A two-minute measurement before any implementation settled the question:
The batch of two costs +52%: the CPU-offloaded experts read almost twice as many weights for two rows. This tells us before implementation that speculation only pays off above 57% acceptance, and at 80% we should expect +8 to +15%, not the 50–80% that a model fully resident in GPU memory would deliver. The subsequent measurement landed exactly in this window. Two minutes of measurement correctly predicted the outcome of two days of implementation. This has since become the documented go/no-go criterion for every additional model.
Three problems, three solutions
1 · Do not reinvent it: upstream had the semantics
Research before coding showed that upstream already contained an MTP path for this exact model family. The local fork had diverged too far to reuse it directly, so its semantics were adopted (graph construction, tensor flow, memory layout) and reimplemented against the local interfaces: around 120 core lines plus a measurement tool, disabled by default.
Draft head: normalization → concatenate with token embedding → projection → attention + MoE block → shared output head2 · The rollback trick for Mamba hybrids
Rejected predictions require state rollback. Attention caches support this by position; this hybrid’s recurrent Mamba state does not (upstream restructures the entire compute core to support it). The solution here: a checkpoint sequence copied before each speculative step and restored on rejection. The same economics, zero kernel changes.
Proof: 299 out of 299 restores with bitwise-identical output values3 · The byte-identity investigation
The speculative output was not byte-identical to baseline. After this project’s four bit-level traps, that was an alarm signal deserving investigation, not a footnote. The decisive experiment: a baseline with an appended dead second row, no speculation logic at all, just a change in batch shape. It too changes the output after approximately 100 tokens. The kernels in this setup simply are not invariant to row count; every emitted token remains the maximum of its own row. Not a logic error, but a documented property.
A mechanical error and a numerical property are different diagnoses, and only one is a bugAcceptance and throughput
Measured with deterministic selection, alternating arms with cooldowns (the card throttles at its power limit), using the median of multiple runs:
| Text type | Acceptance | Tokens/step | Without MTP | With MTP | Δ |
|---|---|---|---|---|---|
| Code generation | 97.7% | 1.98 | 93.4 | 106.9 | +14.5% |
| Lists / facts | 94.2% | ||||
| Wikitext continuation | 88.2% | 1.88 | 95.1 | 99.7 | +4.9% |
| Chat responses | 79.9% | 1.79 | 95.8 | 97.8 | +2.0% |
The cost breakdown confirms the calibration: of 18.3 ms per speculative step, approximately 14.8 ms goes to verifying the batch of two (the partial-offload surcharge), just 1.4–1.6 ms to the draft head itself, and less than 1 ms to checkpoint mechanics. The predictor is cheap; verification pays the offload tax.
A comparison of the two approaches on the same card:
| Approach | Predictor | Draft cost | Acceptance | Result |
|---|---|---|---|---|
| Draft model (classic) | Unrelated 1B model | Full forward pass | Mediocre | +0.9% |
| MTP (built in) | Jointly trained layer 52 | 1/52 of the model | 80–98% | +2 to +14.5% |
Conditionally worthwhile, with a quantified ceiling
Hypothesis confirmed, upper bound measured
The built-in predictor works: +14.5% on code is real, reproducible, and comes without quality loss; the verification logic guarantees equivalent outputs. For code-heavy workloads with deterministic selection, server integration already pays off on this setup.
The ceiling is equally clear: the +52% batch surcharge from partial offload limits the gain. According to the calibration, the same head on a model fully resident in GPU memory would be worth 1.5–1.8×. That was the prediction. It was subsequently measured. The result is in the addendum, and it is more instructive than a confirmation.
Transferability: the GLM and DeepSeek families contain the same prediction block. As pure transformers, they even support simpler position-based rollback, eliminating the Mamba workaround. The two-minute calibration determines whether implementation is worthwhile for each model before work begins.
The lesson beyond this case: speculation fails not because of the principle, but because of predictor and verification costs. Both can be measured before writing a single line of code. The earlier failure (+0.9%) and today’s success (+14.5%) follow the same formula; only the inputs differ.
The prediction falls, the benefit doubles
1.5–1.8× falsified +28–33% confirmed across text types
The plan to fit the same model entirely into GPU memory with a smaller quantization ran into a hard structural finding: every provider’s entire quantization ladder bottoms out at 17.5 GiB, even at its most aggressive level. One model dimension is not divisible by the K-quant block size, so the expert tensors always fall back to the coarser 4-bit format during quantization. A quantization that fits cannot exist with the current tools. Instead, a second model with a built-in prediction head was ported (Qwen3.5-9B-MTP, lossless 8-bit quantization, 9.3 of 16.3 GiB, with no tensor in system RAM).
| Text type | Acceptance | Without MTP | With MTP | Δ fully resident | Δ partial offload |
|---|---|---|---|---|---|
| Code generation | 93.9% | 56.7 | 75.4 | +33.0% | +14.5% |
| Wikitext | 90.0% | 56.5 | 73.6 | +30.1% | +4.9% |
| Chat | 86.8% | 56.5 | 72.6 | +28.4% | +2.0% |
Why 1.5–1.8× still did not materialize: The prediction rested on the on-paper premise that a batch of two is almost free in memory-bound decoding. That does not hold on this architecture: even with full residency, the second row costs +31.8%, because the matrix-vector kernel processes the second column at far from zero cost (the premise holds on tensor-core GPUs, not GCN). Together with prediction overhead, the ceiling is approximately 1.35×, even with perfect acceptance. The two-minute calibration then predicted the three final results to within 2%: 1.27/1.32/1.29 predicted, 1.28/1.33/1.30 measured.
A notable detail: acceptance did not suffer with the different model; it was actually higher on prose and chat. The correctness chain also held unchanged (299/299 bit-exact restores, with divergence from baseline only at a near tie with a margin of 0.022).
This identified the next lever, once again one level deeper: the gain is limited not by speculation, but by the matrix-vector kernel’s weakness with multiple columns, the same bottleneck that slows batched serving. Now fixed: Diagnosis showed an instruction-bound kernel, not a bandwidth-bound one. The weights were not being loaded twice, but every column required a full set of compute instructions, and the multi-column case lost all specialized paths. After extending amortization to the column dimension, the MTP gain reached +44.3% on code (up from +33.0), serving with four concurrent requests improved by +25%, and the batch-1 path was demonstrably unchanged. The series remains consistent: every answered question precisely identifies the next. The next one is a mini-GEMM between matvec and matmul, because without tensor cores the marginal column still costs approximately 20–30% instead of approximately zero.