Model benchmarks

Local model performance, measured on one machine. How it's measured

One row per model, using its fastest config (chat generation where measured, otherwise llama-bench generation at empty context). Chat t/s and TTFT come from the chat benchmark: the same eight real prompts sent to every engine through its HTTP API, one at a time, so they compare across engines. PP, TG, VRAM and tok/J are means of repeated llama-bench runs, which only llama.cpp has.

Model	Engine	Size	Config	Ctx	Chat t/s ↑	TTFT ms	PP t/s	TG t/s	TG t/s deep	VRAM	Tok/J
Qwen3.8 27B Q4_K_M Q4_K_M · dense	llama.cpp	17.44 GB	32k ctx, f16 KV	32k	30.8	220	965	32.4	30.3 @16k	17.3 GB	0.13
Qwen3.8 27B W4A16 AutoRound W4A16 · dense	vLLM	15.81 GB	64k ctx, DFlash2 speculative decoding	64k	146	137	—	—	—	—	—
Gemma 3 4B IT Q4_K_M Q4_K_M · dense	llama.cpp	2.49 GB	32k ctx, f16 KV	32k	—	—	8,196	180	163 @16k	3.6 GB	0.87

Chat generation speed

Decode tokens per second in the chat benchmark, default sampling, every engine. Click a bar to open the model.

Generation speed (llama-bench)

Tokens per second at empty context. Click a bar to open the model.

Speed vs weight size

llama-bench generation. Bigger weights mean more memory traffic per token.