Research
I investigate how AI systems fail and what it takes to make them reliable. Notes on adversarial evaluation, inference correctness, and the engineering behind local AI systems.
One system. Fourteen investigations.
The main series follows one independent engineering project: a llama.cpp fork and a Rust serving layer for AMD gfx906 GPUs. The investigations connect kernel performance, speculative decoding, state correctness, and concurrent serving. Limits exposed in one experiment often become the question for the next.
The reports cover different models, one- and two-GPU setups, and successive versions of the fork. Later measurements sometimes revise earlier explanations. The three inference papers develop findings from this work; the image-compression whitepaper is a separate project.
Project overview, contributions, and benchmark methods →Additional measurements include 750 tokens/s aggregate serving and +84% Lightning decode at 32k. A separate Lightning DSpark experiment reached 163 tokens/s on short code. Each measures a different workload; configurations and evidence are listed on the project page.
Selected investigations
Fourteen Hours to 100 Tokens per Second
Following 35B-model inference through kernels, state management, and cross-device copies.
Read the case study → 02 · CorrectnessWhen a passing test misses the state bug
A reassuring perplexity check, a missing token history, and the test that exposed it.
Read the investigation → 03 · ServingThe Layer That Costs Nothing
Measuring the overhead of a Rust serving layer, from single-user requests to concurrent streams.
Read the case study →How to read 55, 100, and 149.8 tokens/s
These headlines describe different experiments. Model, workload, and decoding method change between them.
| Reported result | Configuration | What it establishes |
|---|---|---|
| 55.3 tok/sAugust 2026 | Qwen3.8-27B-Uncensored Q4_0, DFlash2, single-stream code generation on two Radeon Pro VII cards. | Up from 32.0 tok/s in that report's baseline. Exclusive remeasurement: median 55.2 over eight runs. |
| 100 tok/s2–3 September 2026 | Qwen3.6-35B-A3B, Q4_0 / Router257, two Radeon Pro VII cards, decode without a drafter. | Campaign DX moved the fork from 72 to 99 tok/s. The report's follow-up records 100 tok/s after campaign DZ the next evening. |
| 149.8 tok/sOverview updated 4 September 2026 | Qwen3.6-35B-A3B, Q4_0 / Router257, DFlash, selected single-request code peak on two Radeon Pro VII cards. | A peak from a gate sweep. The later selected profile reached about 145 tok/s on code; prose and story were slower than its base profile. |
The 35B results do not measure the same workload as the 27B result. Single-stream generation, aggregate serving throughput, and kernel microbenchmarks are kept separate throughout the series.
Papers 04
Independent technical reports and a whitepaper. These publications have not been peer reviewed.
- Chained Acceptance Is a Model PropertyPDF · 6 pages ↗
Multi-token prediction and the limits of chained draft acceptance.
- Why Your Prefetch Isn’t PrefetchingPDF · 8 pages ↗
Double-buffered GPU kernels and the gap between intent and execution.
- Speculation Needs RollbackPDF · 4 pages ↗
State management and correctness in speculative decoding.
- Verified Perceptual Image CompressionWhitepaper · PDF · 5 pages ↗
Delivering a per-file quality floor for professional photography.
Case Studies 14
Fourteen reports from the gfx906 project, grouped by their main question. Dates reflect the investigation period or the report's stated as-of date; month-only dates are retained where no day is given.
Performance 07
Which part of inference is actually limiting throughput, and under which conditions does an optimization help?
- Fourteen Hours to 100 Tokens per Second
Tracing kernels, state handling, and cross-device copies moved 35B decode from 72 to approximately 99 tok/s without a drafter.
- Two Retired GPUs, 55 Tokens per Second
DFlash2 raised 27B single-stream code generation from 32.0 to 55.3 tok/s; gains depended on content, context, and concurrency.
- The Kernel That Waited for Itself
Compiler reordering defeated prefetching. Enforcing the load, compute, and store order made double buffering effective.
- Three Models, One Draft Head
Similar raw model throughput hid a 10% serving difference caused by the supplied draft head's chained acceptance.
- Built-in Speculation
A jointly trained MTP head made speculation useful on the tested setup; verification cost still limited the gain.
- Six Hypotheses, Four Refutations, +31% Throughput
Profiling shifted attention from launch overhead to work inside the kernels, including redundant activation loads and occupancy effects.
- The Reckoning
A sweep across 35 comparable models found +5% median decode, larger format-specific gains, losses, and a claim that had to be withdrawn.
Correctness 03
Does the system preserve the model's behavior, and do the tests exercise the state transitions that can fail?
- A Crash in Three of Nine Runs
Repeated runs overturned an initial race diagnosis. A mitigation passed twelve runs, but the underlying cause remained unresolved.
- 480 Billion Parameters on 32 Gigabytes–
CPU expert offload, system RAM, and NVMe enabled the 480B model on two 16-GiB GPUs; batch-1 evaluation exposed missing token history.
- Every Public GGUF of This Model Is Missing Its Head
Restoring omitted MTP tensors was only the first step: apparently excellent acceptance concealed full-model self-speculation.
Serving 04
How much of the kernel performance reaches the client, across real requests, model setups, and concurrent users?
- The Hunt for 50/60
Lifecycle bugs escaped single-request tests. Fixes helped reach 60.67 tok/s across three streams; solo code generation remained at 45.80.
- Running Qwen3.8 on a Deprecated GPU
Missing rocBLAS kernels explained a compatibility barrier; the report documents a working 27B setup and its remaining limits.
- The Layer That Costs Nothing
A Rust server kept measured overhead low, while ABI layout checks caught two drifts that could have corrupted memory.
- The Gap Between One and Many
Small batches needed their own kernel path. Mini-GEMM reduced kernel time at 12 columns; separate serving tests measured client throughput.
Current research
When a passing test misses the state bug
A batch-512 perplexity check did not expose missing token history in a Flash-Next port. A batch-size-1 evaluation did. Read the investigation →
-
Fellowship applications
I passed the initial application screening for the Cambridge ERA:AI Fellowship and the Anthropic Fellows Program. Both selection processes are ongoing.