The layer that costs nothing
After 60 commits of kernel work, the card delivered more performance than its server could serve. So fork-serve was born: a Rust server whose design goal is measurable invisibility. Roughly level in single-user operation, slightly ahead under concurrency, and with classes of errors ruled out by the language. Getting there involved a failed experiment, a machine-checked memory-layout proof that prevented silent corruption twice, and a leak hunt that ultimately found a bug in AMD's runtime.
llama-server with 8 users
alongside three running streams
the layout check
last 120 of 360 requests
Fail first, then draw the boundary correctly
Rust kernels: 19× slower, discarded
The Rust question began with an obvious ambition: could the GPU kernels themselves be written in Rust? An experiment using the Burn/wgpu ecosystem was built and measured: 19 times slower than the hand-optimized HIP kernels, and limited to half precision. The verdict came early and clearly: on this architecture, the kernels stay in C++/HIP.
But that failure put the boundary in the right place. Everything above the kernels, process lifecycle, memory ownership, concurrency, HTTP, streaming, is precisely where Rust's guarantees help and C++ serving layers accumulate bugs. That is how fork-serve emerged: the kernels stay where they won; the serving layer moves to where the language rules out classes of errors.
Four decisions underpinning the server
1 · Handwritten bindings with machine-checked layout
Instead of generated bindings: ~220 handwritten lines, but a checking program measures every C structure on both sides of the language boundary, field by field and offset by offset. That sounds pedantic until it pays off: the check has caught real drift twice, most recently a parameter structure that grew by eight bytes after the fork added a field. Without the check, this would have been silent memory corruption; with it, the mismatch was corrected during the build process.
An ABI error costs days in the debugger, or one line in a verification log2 · Ownership instead of discipline
Every GPU resource (model, context, sampler, batch) lives in an RAII wrapper: cleanup is a matter of the type system, not convention. Cancellation is not a special case: when the client disconnects, its receiver disappears and the sequence cleans itself up. The endurance test supports the model: 400 requests over ten minutes, memory after warmup flat.
RSS trajectory: 765 → 790 MB during warmup, then +96 kB over 9.3 minutes3 · The scheduler: one thread owns the engine
Continuous batching without locks: a single thread owns the GPU engine and builds one shared batch per step from all active requests. Ongoing outputs come first, then chunks of incoming prompts within a fixed token budget so a long prompt cannot block everyone else. Requests join between steps; overload is explicitly rejected (HTTP 429) rather than hidden in a buffer.
Up to 8 sequences per step · 512-token prefill budget · Timeout cleanup4 · The leak hunt that reached AMD's runtime
The oldest fork-serve story is the best: memory growth of ~200 kB per request looked like a bug in the new server. Bisection ended somewhere else: a leak in the ROCm runtime itself (~370 bytes per graph node per invocation), reproducible in a 60-line proof program without any Rust involvement. The server was innocent; the platform was not.
Sometimes, suspecting your own house leads you to a leak next doorWhat the thin layer delivers
The standard is strict: not what the hardware can do, but what reaches the client. The result is less spectacular than an earlier measurement in this project suggested (see the discussion below): fork-serve sits slightly behind llama-server for a single user and 3–4% ahead from four concurrent requests onward. The serving layer therefore has almost no measurable cost. That was the goal all along.
| Concurrent users | fork-serve | llama-server | Δ |
|---|---|---|---|
| 1 | 110.3 | 113.2 | −2.6% |
| 4 | 240.8 | 233.3 | +3.2% |
| 8 | 307.6 | 295.5 | +4.1% |
The second practical result does not appear in throughput tables: a user joining alongside three running streams gets their first token after 50 milliseconds and then streams at a stable 22.5 ms per token: multi-user operation that feels like running alone. The speculative mode from Part 2 is also included as a Rust port, verified through counter equality with the C++ reference tool (83.7 tokens/s in code mode).
There is also a revised verdict with a clearly bounded scope: HIP graphs, disabled in Part 1 because of performance and a leak, provide in multi-sequence operation +38%, and the leak does not reproduce under the exact historical trigger pattern (360 requests in waves of four, measured independently twice, memory flat). The default now depends on operating mode: enabled for batching, disabled for single-token decode.
A mixture of llama.cpp and vLLM?
The short answer is yes, in spirit, but the analogy has clear limits. The stack takes llama.cpp's kernels and formats, plus 60 commits of custom gfx906 work, and adds vLLM's central idea: requests share every compute step instead of waiting in line. But actual vLLM does more: paged KV cache, prefix caching, 64+ users. Their sweet spots therefore differ, and both have been measured on this card:
| fork-serve + fork | llama-server | vLLM-gfx906 | |
|---|---|---|---|
| 1–8 users (7B) | 247–320 t/s, TTFT 50 ms | 170–303 t/s | only worthwhile at scale |
| 64 users | not designed for this (8 slots) | ~390 t/s | |
| Model formats | any GGUF, MoE, offload | any GGUF | AWQ/GPTQ, narrow set |
| Speculation | Native MTP (greedy) | Draft models | unavailable on gfx906 |
| Serving-layer loss | ≈ 0 (measured) | ~25% at c=4 | low overhead |
A more precise description than “mixture” is therefore: llama.cpp's world, taught the one vLLM concept it lacked most, tailored to a realistic operating point for one 16-GB card: a handful of concurrent users, every model format, low latency. For 64 users, the vLLM container is the better choice; for a home or small-team server, this setup delivers more than either original.
Honest gaps: no chat templating or OpenAI-compatible API, no cross-request prompt caching, no grammars, sampling without penalties, and speculative mode limited to deterministic, sequential operation. The next major building block for scale would be a paged KV cache. That really would mean rebuilding vLLM, and the container is probably the better option for that. This server occupies a different niche, supported by measurements.
What the custom server demonstrated
First, the boundary hypothesis: kernels in C++, serving layer in Rust, each side at its measured optimum. Second, that a serving layer can have effectively no measurable cost: the entire advantage at four users is not a kernel miracle but absent overhead. Third, that discipline scales: the layout check, counter equality for the speculative port, and the four-prompt test against sequence contamination each caught at least one issue that would otherwise have silently reached production.
And fourth, looking back over the series: the kernel work in Parts 1–3 makes each compute step faster, continuous batching fills each step with multiple users, and the thin layer delivers almost all of it. The layers multiply: from the project's starting point (88.6 tokens/s for one user) to today: 320 tokens/s delivered to eight concurrent users, on the same card.
The series began with a card that computed incorrectly. For now, it ends with a server that computes correctly, serves quickly, and has error classes that are measured rather than assumed.