← Benchmarks

How it's measured

Every number on this site comes from a scripted run on one machine. This page explains what each run does, what the numbers mean, and what they don't cover.

The machine

• NVIDIA GeForce RTX 3090 with 24 GB of VRAM at a 250 W power limit (driver 595.99.02, CUDA 13.2); AMD Ryzen 5 3600 6-Core Processor with 6 threads; 16 GB of RAM.
• Engine builds: llama.cpp@91f6a6cf3.

Speed runs

Speed comes from llama.cpp's own benchmark tool, llama-bench, running directly on the GPU with no server in the loop. By default each config is tested with:

• Prompt processing: a 512-token prompt.
• Generation: 128 new tokens.
• Context depth: 0, 4k and 16k tokens already in the cache, to show how speed falls as a conversation grows. Depths that don't fit a config's context length are skipped.
• Repetitions: 5 of each test, with a 3-second pause between tests. Tables show the mean and standard deviation.

Every setting is explicit

Each config's exact launch command is on its model page. Settings that an engine would otherwise pick for itself are always pinned: --fit off, parallel slots (-np), --cache-ram 0, flash attention, KV-cache precision, batch and micro-batch size, CPU threads, MoE expert offload (-ncmoe) and how weights are loaded.

Configs are hash-locked: changing any setting creates a new config instead of quietly rewriting published results.

A clean GPU for every run

Before a run starts, the lab takes an exclusive lock on the GPU, stops the chat model this machine hosts, and waits until less than 300 MiB of VRAM is in use and the GPU has cooled to 45 °C or below. The chat model restarts when the run ends, even if the run fails.

Power and efficiency

• GPU telemetry (VRAM, power, temperature, utilization and throttle reasons) is sampled every 100 ms through NVIDIA's NVML.
• Power for a test is the mean GPU power over that test's samples where utilization was at least 50%.
• Tokens per joule is tokens per second divided by that power. Only GPU power counts: CPU power isn't readable from inside the container.
• Peak VRAM reflects llama-bench's cache sizing for its tests (prompt, generated tokens and depth), not a server holding a full context window.

Throttling

A run is flagged as throttled if the GPU reports hardware slowdown, thermal slowdown or a hardware power brake at any point. Running into the card's configured power limit is expected and doesn't count. Throttled runs stay visible with a tag, but they never feed the headline numbers.

Reproducibility

Every run records a hardware snapshot (GPU, driver, CUDA, power limit, CPU, RAM and kernel), the llama.cpp commit, and the exact command. Each run has its own page with everything that was published, including the benchmark settings and a telemetry summary.

What these numbers don't show

• They describe this one machine, not every RTX 3090.
• They measure one request at a time with synthetic prompts, not multi-user serving throughput or end-to-end chat latency.
• Output quality (perplexity, KL divergence) and task evals aren't measured yet.