Local Inference

Local Inference

Local Inference is my lab for running open-weight language models on a single RTX 3090. I tune each model's settings (context length, KV-cache precision, offload), then rank them on the work I actually want done: writing code, driving tools, reasoning. Speed counts here because it buys thinking time, not because a quick answer is worth more — every answer has to finish inside a token allowance taken from that config's own measured speed. Every number comes from a scripted, hash-locked config, so results are reproducible.

How it's measured →

Fastest chat generation
146 t/s
Qwen3.8 27B W4A16 AutoRound · vLLM
Generation at 16k context
163 t/s
Gemma 3 4B IT Q4_K_M · llama-bench
Best chat efficiency
0.57 tok/J
Qwen3.8 27B W4A16 AutoRound · vLLM
Benchmarked
3 models
6 configs

Benchmarks

Every model and config ranked for coding work, and the speed behind that ranking: chat generation on every engine, generation as the context fills, VRAM, power and efficiency, with the exact launch command for each.

View benchmarks →