Independent project · AI systems engineering
LLM inference,
down to the metal.
Pushing AMD gfx906 through custom GPU kernels, speculative decoding, and a Rust serving layer. From small-batch throughput to 35B-model inference.
Measured performance highlights
Selected measured peaks and gains from different workloads, not one combined benchmark. Configurations and comparisons ↓
- Platform
- AMD gfx906 / ROCm
- Stack
- C++ · HIP · Rust · Python
- Focus
- Correctness & performance
The problem
I wanted to understand what a constrained local inference setup could actually do. Getting a model to load was only the beginning. Optimized kernels could produce wrong answers, extra state copies could erase a speedup, and a successful benchmark could miss the path used in real decoding.
The project grew from a TurboQuant-related llama.cpp fork into an investigation of the full stack: quantized GPU operations, graph execution, speculative decoding, expert offload, and serving. The experiments discussed here use one or two gfx906 cards. They are separate from my wider 23-GPU infrastructure project.
What I worked on
Kernel correctness
Investigating register aliasing in inline assembly, quantization errors, unsafe prefetch paths, and dispatch decisions. Connecting a failure to a small operation and a regression test.
Small-batch performance
Developing and testing specialized quantized matrix kernels, including double-buffered shared-memory staging. Measuring memory traffic, register pressure, and the cost of different batch shapes.
End-to-end serving
Building fork-serve, a Rust layer around the fork's libllama, with scheduling, caching, and integrations for MTP and DFlash. Following state and overhead across the boundary between server and runtime.
Evidence and corrections
Keeping benchmark data, profiling notes, negative results, and revisions to earlier explanations. Separating a faster kernel from a faster request, and a working test from a correctness guarantee.
Behind the numbers
| Workload | Baseline → result | What changed |
|---|---|---|
| 35B-A3B · DFlash code peak | 149.8 peak | DZ.7: gate 200:4, previous ring path. A selected peak, not the final profile or an average. |
| 35B-A3B · selected code profile | 97 → 145 +49% | EA: base/Lookup versus DFlash, gate 250:4:64. Two GPUs, 256 output tokens, temperature 0. |
| 35B-A3B · no-drafter decode | 72 → 99 +38% | DX start to end: combined kernel, state, and cross-device-copy work within the fork. |
| 35B-A3B · DFlash at 6k | 37 → 82 2.2× | DX start to end. A context-specific development comparison, not current-upstream performance. |
| Lightning 30B-A3B · DSpark code peak | 163 short code | 30 August, DL: two MI50 GPUs at 190 W each, 16k configured context, NMAX 7, PMIN 0.7. Longer code: 112; prose: 140–143; story: 86. Different model from the 35B result. |
| Qwen3.5-35B-A3B APEX Mini · prefill | 120.98 → 939.03 7.76× / +676% | 15 August comparison TSV: pp512, one MI50 at 150 W, identical flags, warm-up excluded. The report attributes the gain to a Q2_K register-spilling fix. Older model metadata is inconsistent; exact quantization provenance remains to be reconciled. Not a decode gain. |
| Qwen3-8B Q4_0 · two independent services | 375 → 750 2.0× aggregate | 24 August, BY.1: one versus two MI50 cards; eight streams per card, 256 prompt and 256 generated tokens. Two-card decode is 377 + 373. The separate 1,150 tokens/s pipeline figure includes prompt processing and is not decode-only. |
| Lightning 30B-A3B · decode at 32k | 49.4 → 91.0 +84% | 23 August, BM.2: upstream versus fork, one gfx906 card at 190 W, Q4_0, q8 KV, six expert blocks on CPU, b2048/ub512, cooling gate below 50°C. Fork decode falls from 95.7 at 512 tokens to 91.0 at 32,768. |
| Lightning · agent follow-up prefill | 706 → 49 ms 93% lower latency | 30 August, DM.1: turn four of a controlled five-turn test with opt-in DFlash prefix resume. Best reported effective agent-loop throughput: 97.8 tokens/s. This is not a 14× speedup of the entire agent task. |
| Qwen2.5-1.5B Q4_0 · single-stream decode | 234.24 → 327.43 +39.8% | 15 August comparison TSV: tg64, one MI50 at 150 W, warm-up excluded. Fork standard deviation: 0.20 tokens/s. A smaller-model result, not comparable to 30B/35B peaks. |
| Qwen3-8B · decode at 32k | 14.9 → 60.5 4.1× | BM: q8 KV, upstream b10524 versus fork. Avoiding the baseline's full-KV conversion cost. |
| Qwen3-8B · batch 16 | 329 → 586 +78% | BM: f16 KV, upstream b10524 versus fork. Aggregate decode, not one user's generation speed. |
| Mistral-Small-24B Q4_0 · batch 16 | 126 → 224 +78% | BC: db-gemm off versus on; context 4096. Kernel-path ablation, with 45–60-second pauses between runs. |
Hardware. The 35B results use two Radeon Pro VII cards with 16 GiB each, approximately 190 W per card, and GPU-resident Q4_0 / Router257 weights. The BM upstream comparison used gfx906 at 190 W. These are separate experiments from the 23-GPU cluster.
Workload matters. In the later EA comparison, the 35B DFlash profile reached 145 tokens/s on code, but 90 on prose and 88 on story against approximately 100 for the base profile. The 149.8 peak is retained as a measured result, not presented as the fastest setting for every prompt.
Scope. The 4.1× result repairs a specific long-context dispatch disadvantage. It does not imply 4.1× faster inference in general. Likewise, the B16 figures are aggregate throughput. The broader 35-model comparison from 15 August had a +5.0% median decode gain, with one slightly negative point estimate.
Evidence: DX-35b-hundert-02-09-2026.md (DX final table, DZ.7 and EA); BENCHMARKS.md (BM, BM.2, BC and BY.1); DL-kampagne-30-08-2026.md (PMIN sweep); DM-kampagne-30-08-2026.md (DM.1 resume); bench-harness/results/vergleich-upstream-2026-08-15.tsv (APEX Mini and Qwen2.5-1.5B); MASTERPAPER-EVIDENZREGISTER.md (APEX provenance caveat). These records are in the private project archive. Full run artifacts are not yet published, and results have not been independently replicated.
The result I find most useful
A perplexity check at batch size 512 looked reassuring. A later batch-size-1 test exposed missing token history across graph boundaries in the Flash-Next port. Both earlier paths could pass the comparison without exercising the state transition that mattered.
Research noteWhen a passing test misses the state bugRead the investigation →Scope, credit, and open work
This work builds on llama.cpp, community gfx906 kernels, TurboQuant, DFlash, and upstream architecture and cache work. My contributions include debugging, kernel adaptations and development, integration, serving, and evaluation. I did not invent the underlying quantization or speculative-decoding methods.
An intermittent multi-GPU expert-cache failure remains an open investigation. Successful mitigation runs are not proof of a root cause or production readiness. Some earlier explanations for slow MTP were revised after additional measurements.
The code archive and full reproduction package are not published here. Build portability, pinned run manifests, and independent reproduction are still outstanding. AI assistance was used during development; it is not independent validation.
Foundations: llama.cpp · TurboQuant · DFlash