hosted/ai

models

Throughput | tokens/sec

generation · depth 0 · llama-benchy

Generation t/s vs context depth

how far throughput falls as the prompt grows

bench results

software stack

hardware

how the stack fits together

llama-server: The inference server from llama.cpp. It is the process that actually runs the models: loading weights, managing the KV cache, and exposing an OpenAI-compatible chat endpoint. This is only capable of serving exactly one model at a time, leading to the need for a router.
llama-swap: The router sitting in front of llama-server. Llama-swap presents a single OpenAI-compatible API automatically switching the resident model to match incoming requests. This is what allows for multiple models on a single GPU.

testing methodology

The goal is a fair, repeatable read on how a given model behaves on this hardware. Every statistic here is measured against a live llama-swap endpoint with llama-benchy rather than taken from marketing. Each figure is a mean across repeated runs, reported with its standard deviation so the spread is visible instead of hidden.

Runs are warm: the model is already resident before timing starts, so a cold llama-swap load is never counted. Each model is then swept across a range of context depths, the amount of conversation already in the KV cache before the measured request, because throughput on a small card degrades as that cache grows.

Two kinds of test are recorded at every depth:

pp2048 - prefill: Processing a 2048-token prompt. This is compute-bound and runs in the thousands of tokens/sec; it determines how long you wait before the first word appears.
tg32 - generation: Generating 32 tokens. This is memory-bandwidth-bound and runs far slower; it is the number that decides whether a model feels usable to type at.

And five metrics per test:

t/s and peak t/s: Mean and best sustained throughput. Higher is better.
ttfr: Time to first response - the wait before any output arrives. Lower is better.
est_ppt: Estimated prompt-processing time, the share of that wait spent on prefill.
e2e_ttft: End-to-end time to first token, measured from the client including request overhead.

The VM and models are: checking…