How it's measured

Every number on this site comes from a scripted run on one machine. This page explains what each run does, what the numbers mean, and what they don't cover.

The machine

NVIDIA GeForce RTX 3090 with 24 GB of VRAM at a 250 W power limit (driver 595.99.02, CUDA 13.2); AMD Ryzen 5 3600 6-Core Processor with 6 threads; 16 GB of RAM.
NVIDIA GeForce RTX 3090 with 24 GB of VRAM at a 250 W power limit (driver 595.99.02, CUDA 13.2); AMD Ryzen 5 3600 6-Core Processor with 6 threads; 24 GB of RAM.
The host caps the GPU at 250 W and holds its core clock at or below 1,625 MHz from boot, a conservative power setup. Prompt processing gives up a few percent to it; generation speed doesn't change.
Engine builds: llama.cpp@91f6a6cf3, vllm@bae2023ff.

Chat benchmark (all engines)

Every config, on every engine, answers the same chat requests through its own HTTP API, the way a chat client talks to it. These are the numbers to compare across engines.

Prompts: eight real chat prompts, each answered with up to 1,024 tokens, one request at a time, streamed.
Warm before timing: the server boots, then answers all eight prompts once, and those answers are thrown away. vLLM compiles kernels the first time it boots, so nothing is timed until the server is warm.
Two passes: default sampling (temperature 1.0, top-p 0.95, top-k 20, the same for every model) and greedy decoding (temperature 0). Pages lead with default sampling.
Thinking off, so every request times the answer itself. The chat this machine hosts runs with thinking on.
Time to first token (TTFT): from sending the request to the first streamed token.
Time per output token (TPOT) = (last content chunk − first content chunk) ÷ (completion tokens − 1). The token count comes from the server, never from counting chunks: speculative decoding sends several tokens in one chunk.
Decode speed = 1,000 ÷ mean TPOT in milliseconds. End-to-end speed also counts the wait for the first token.
No cached prompts: each request gets its own cache salt on vLLM, and llama.cpp runs with prompt caching off. A run that reads anything from a prefix cache fails.
Speculative decoding drafts several tokens ahead and checks them in one step. How many are accepted depends on the text, so other prompts can run faster or slower; the accepted-per-step column shows the average.
VRAM: vLLM reserves its whole KV-cache pool when it starts, so its peak VRAM is what it preallocates, not what the requests used.
The command line and settings the server actually ran with are recorded with each run, without local paths.
Expect a few percent of variation between repeated runs, and up to about 5% for greedy decoding with speculation.

Protocol and prompts adapted from syv-ai/qwen38-27b-rtx3090 (Apache-2.0).

Speed by context depth (llama-bench)

llama.cpp configs also run llama.cpp's own benchmark tool, llama-bench, directly on the GPU with no server in the loop. It shows how speed changes as the context fills, which the chat benchmark doesn't. By default each config is tested with:

Prompt processing: a 512-token prompt.
Generation: 128 new tokens.
Context depth: 0, 4k and 16k tokens already in the cache, to show how speed falls as a conversation grows. Depths that don't fit a config's context length are skipped.
Repetitions: 5 of each test, with a 3-second pause between tests. Tables show the mean and standard deviation.

Every setting is explicit

Each config's exact launch command is on its model page. Settings that an engine would otherwise pick for itself are always pinned. For llama.cpp: --fit off, parallel slots (-np), --cache-ram 0, flash attention, KV-cache precision, batch and micro-batch size, CPU threads, MoE expert offload (-ncmoe) and how weights are loaded. For vLLM: context length, KV-cache pool size, request slots, speculative decoding and prefix caching, with the serving scripts pinned to one commit.

A benchmark never chooses how a model samples: those fields are stripped from every request, so what runs is what the config was launched with. Each run records that setting as the server reports it. llama.cpp is asked directly; vLLM does not expose it, so its record comes from the model's own generation config plus any override in the launch.

Configs are hash-locked: changing any setting creates a new config instead of quietly rewriting published results.

A clean GPU for every run

Before a run starts, the lab takes an exclusive lock on the GPU, stops the chat model this machine hosts, and waits until less than 300 MiB of VRAM is in use and the GPU has cooled to 45 °C or below. The chat model restarts when the run ends, even if the run fails.

Power and efficiency

GPU telemetry (VRAM, power, temperature, utilization and throttle reasons) is sampled every 100 ms through NVIDIA's NVML.
Power for a test is the mean GPU power over that test's samples where utilization was at least 50%.
Tokens per joule is tokens per second divided by that power. Only GPU power counts: CPU power isn't readable from inside the container.
Peak VRAM from llama-bench reflects its cache sizing for its tests (prompt, generated tokens and depth), not a server holding a full context window. The chat benchmark measures a real server instead.
In the chat benchmark, power and tokens per joule for each pass come from NVIDIA's cumulative energy counter over that pass.

Capability evals

Speed alone doesn't say whether a model can do the work, so each config is also scored on what it solves. Every benchmark runs against the config's own server, with its own settings, and reports the percent of tasks solved with a standard error.

Quick tier, run for every config: LiveCodeBench (100 problems), BFCL (400 cases), GPQA Diamond (198 questions) and AIME 2025 (30 problems, 4 attempts each). Each task gets 5 minutes.
Deep tier, run only for configs worth the overnight time: SWE-bench Verified (30 issues, solved by an agent) and Terminal-Bench (30 tasks). Each task gets 20 minutes and runs alone.
Only configs with all six benchmarks are ranked. The rest are listed by their quick-tier score.
Each config is evaluated exactly as it serves chat, including thinking where it has it. Thinking tokens come out of the allowance, so reasoning is paid for rather than free. Speed runs don't involve thinking: llama-bench times fixed token counts without a chat template, and the chat benchmark turns thinking off.
AIME 2025 and GPQA Diamond are graded by the lab: the boxed integer, and the last "Answer: X" line of the answer. LiveCodeBench and BFCL run their own published code for prompts and grading (LiveCodeBench at a pinned commit, bfcl-eval 2026.3.23); the lab only passes the conversation to the model and back.
LiveCodeBench uses the 100 newest problems of its v6 release, from 15 February to 6 April 2025. BFCL uses 80 cases from each of five categories: simple, multiple, parallel, parallel-multiple and multi-turn.
BFCL calls tools natively when the model's server can, and otherwise uses BFCL's prompting mode, which lists the functions in the system prompt. Every result says which mode ran, and two rows only compare directly when their modes match: prompting mode asks a model to imitate a format it was never trained to emit, which costs it most on the parallel and multi-turn cases.
Code a model writes during a benchmark runs in a locked-down container with no network access, separate from the model server.

The thinking allowance

Speed matters here because it buys reasoning, not because a fast answer is worth more. Each task gives a config a token budget instead of a stopwatch: what it could generate within the time limit at its own measured speed, after reading the prompt.

Allowance = (time limit − time to read the prompt) × generation speed at that prompt's depth, capped by the context left.
Both figures come from the config's published speed run, so the budget is fixed before the benchmark starts and never drifts with load.
A config with no depth ladder — every engine but llama.cpp, whose llama-bench sweeps context depth — is sized from the chat benchmark's generation speed at an empty context, applied at every depth. Generation slows as context fills, so deep in a long context that budget is generous.
Thinking tokens count against it. An answer that doesn't finish inside its allowance is wrong.
A task that takes several requests, such as a multi-turn BFCL case, shares one time limit. Each request is charged for the prompt tokens the server hadn't already cached and for its output, at the measured speeds, and the next request gets what's left.
A faster config gets more room to think in the same 5 minutes; a model that finishes early gains nothing from extra speed.

The coding-work score

Coding counts 50%, agents and tools 30%, reasoning 20%.
A category's score is the mean of its benchmarks, and the total combines the three as a weighted geometric mean, so being weak in one area pulls the score down rather than averaging out.
The ± figure is one standard error, carried through from each benchmark's own error.
Configs whose scores are within one combined standard error share a rank, shown as "=2".
Issues fixed per night = 8 hours ÷ the average time per SWE-bench issue × the share it resolved.

Throttling

A run is flagged as throttled if the GPU reports hardware slowdown, thermal slowdown or a hardware power brake at any point. Running into the card's configured power limit is expected and doesn't count. Throttled runs stay visible with a tag, but they never feed the headline numbers.

Reproducibility

Every run records a hardware snapshot (GPU, driver, CUDA, power limit, CPU, RAM and kernel), the engine build (the llama.cpp commit, or the vLLM version with the serving scripts' commit and patches), and the exact command. Each run has its own page with everything that was published, including the benchmark settings and a telemetry summary.

What these numbers don't show

They describe this one machine, not every RTX 3090.
Speed runs measure one request at a time: llama-bench with synthetic prompts, the chat benchmark with eight real ones. Neither is multi-user serving throughput.
The benchmarks are small and use fixed subsets: 30 tasks carry roughly ±9 points of error near 50%, which is what the error bars and shared ranks are for.
Each config is benchmarked with one sampling setup, the one it serves chat with. Another setting might score differently.
Models may have seen these public benchmarks during training, which would flatter them. The LiveCodeBench problems date from early 2025, so models trained after that are the most likely to have seen them.
Only configs worth the overnight time get the deep tier, so most are ranked on the quick tier alone.
Output quality (perplexity, KL divergence) isn't measured yet.