Building a 23-GPU local LLM inference cluster
A controlled environment for running, comparing, and evaluating models locally—built from 23 AMD MI50 GPUs across multiple nodes on the ROCm ecosystem.
The problem
Cloud APIs are convenient, but they are a weak foundation for experiments that need control over model versions, runtime configuration, data locality, and repeated measurements. The goal of this cluster is not maximum benchmark theatre. It is a local test environment whose runs can be traced, compared, and repeated.
Architecture
The system separates four concerns:
- Compute: 23 AMD MI50 GPUs distributed across multiple nodes.
- Runtime: the ROCm ecosystem provides the hardware execution layer for local inference.
- Measurement: a reusable Python benchmark suite starts runs with explicit configurations and captures their outputs.
- Persistence: SQLite stores run metadata and measurements so results remain queryable after the terminal session is gone.
What the benchmark harness records
A useful result needs enough context to be reproduced. Each run is designed around a stable record: model and runtime configuration, node and device allocation, workload parameters, timestamps, completion state, and the resulting measurements. That turns an isolated test into comparable evidence.
Engineering constraints
- Heterogeneous systems: multi-node operation makes configuration drift visible quickly.
- Memory limits: model fit, quantisation, and allocation strategy matter as much as raw compute.
- Repeatability: a number without its configuration is not a benchmark.
- Observability: failed and partial runs need to be recorded, not silently discarded.
Why this matters for AI security
A local cluster provides a controlled place for adversarial evaluation: sensitive test corpora remain local, model and runtime versions can be pinned, and defensive changes can be replayed against the same attack set. The infrastructure is therefore part inference lab, part security test bench.