architecture

The whole system, from bare metal up through the models being served, all built from a multi-role Ansible playbook. The Proxmox host passes a GPU to a guest Ubuntu VM; inside that VM, llama-swap sits in front of llama.cpp swapping the resident model on demand. A separate machine on the LAN drives llama-benchy against the llama-swap endpoint to produce the numbers shown across this site.

system diagram

Local AI system architecture PROXMOX VE HOST UBUNTU SERVER VM DOCKER Remote PC llama-benchy RTX 4060 8 GB VRAM llama-swap model router llama-server llama.cpp GGUF weights one model resident VFIO passthrough
request / inference path VFIO GPU passthrough

the layers

Proxmox VE: hosting hypervisor
The bare-metal hypervisor. It owns the physical hardware and runs the Ubuntu guest as a virtual machine, keeping the AI box isolated from everything else on the network.
VFIO passthrough: GPU passthrough
Proxmox passes the entire GPU to the guest VM via Virtual Function I/O (VFIO). The VM sees the card as if it were physically installed, so llama.cpp gets native CUDA performance with no virtualization layer in the compute path.
llama-swap: model router
Presents a single OpenAI-compatible API, automatically swapping the resident model to match incoming requests.
llama-server: inference server
The llama.cpp server that actually runs a model: loading GGUF weights, managing the KV cache, and generating tokens.

infrastructure as code

This box is not configured by hand, rather an Ansible playbook provisions a bare Ubuntu Server install with a passed through GPU and brings it all the way up to a fully featured inference/LLM server. Re-running of the playbook is idempotent, so any unmodified settings are left alone, rather than reset. Every run ends with a verification pass: GPU is live, services are up and reachable, all catalogued models are registered in the API.

What the playbook provisions:

Base OS: packages and tuning
Core packages, timezone, swap behavior and CPU governor. All ensuring a stable and performant platform to build upon.
NVIDIA driver: installed, then pinned
The playbook detects the recommended driver branch, installs the open kernel-module server build alongside the container toolkit.
llama.cpp: built from source
Upstream ships no Linux CUDA binaries, so the playbook compiles a pinned llama.cpp tag with CUDA enabled for this card's architecture. Each build lands in its own versioned directory behind a current symlink, which makes a version bump, or a rollback, a one-line change.
llama-swap: router and model catalogue
One list of models in the playbook drives everything downstream: the GGUF downloads, the generated router config, and the per-model llama.cpp flags. Dropping a model from that list removes it from disk on the next run.
Docker and UFW: runtime and network
The container engine with the NVIDIA runtime wired in, and a default-deny firewall that exposes the inference API to the LAN only.

how it's benchmarked

The numbers across this site come from a separate machine on the LAN running llama-benchy against the llama-swap endpoint on the same OpenAI-compatible API as the models. Nothing is measured on the box itself, such that the benchmark resembles a real client's usage.