The fastest LLM inference engine takes 28 minutes to start

By mid-2026 the open-source inference stack has consolidated around four engines, but the most common question — "which one is fastest?" — still has no clean answer. On a single H100, the top three serve within ~14% of each other, and the ranking flips the moment you change the workload.

Which Inference Engine Is Fastest on H100 in 2026?

There is no single fastest H100 inference engine in 2026 — TensorRT-LLM, SGLang, and vLLM finish within roughly 14% of each other, and the leader changes with workload, concurrency, model architecture, and whether you tolerate a per-model compile step. In a third-party H100 SXM5 80GB test on Llama-3.3-70B-Instruct FP8 at 50 concurrent requests, TensorRT-LLM reached about 2,100 output tokens/s, SGLang about 1,920, and vLLM about 1,850 . The differences are real but narrow — far smaller than vendor headlines imply.

Latency tells the same story. At 10 concurrent requests, p50 time-to-first-token was 105 ms for TensorRT-LLM, 112 ms for SGLang, and 120 ms for vLLM — a 15 ms spread across the field . Memory footprint is effectively a tie: peak VRAM at 100 concurrent requests landed at roughly 78–79 GB on all three engines . Practically, that means your engine choice does not change your hardware budget — if a model fits in one engine's VRAM, it fits in all three.

H100 SXM5 80GB, Llama-3.3-70B-Instruct FP8 (source: Spheron benchmark, 2026).

So why does the "fastest engine" debate persist? Because these numbers describe one workload on one GPU. Rankings reorder when you switch to shared-prefix traffic like RAG or multi-turn agents, where SGLang's KV-cache reuse pulls ahead, or to maximum-throughput batch jobs where TensorRT-LLM's compiled kernels lead — if you can absorb its multi-minute build step. The honest framing for 2026 is operational, not a single benchmark: pick the engine whose trade-offs match your serving pattern, because on raw H100 throughput they are close enough that the deciding factors live elsewhere.

The rest of this guide walks those factors one engine at a time — what vLLM's V1 rewrite actually changed, where SGLang wins by a measurable margin, what TensorRT-LLM's ~28-minute compile buys you, how local hardware behaves, and the throughput levers (FP4, KV reuse, prefill-decode disaggregation) that increasingly matter more than the engine name. The underlying H100 comparison is the baseline we keep returning to.

What vLLM V1 Actually Changes for Your Stack

vLLM V1 is a full rewrite of the engine's CPU-side scheduling that became the default and is now the only option — so if you run vLLM in 2026, you are running V1 whether you opted in or not. The alpha was announced on January 27, 2025, V1 became the default in mid-2025, and the legacy V0 code path has since been fully removed . Migration is forced, not optional; plan a pin-and-test pass before upgrading to recent releases.

The headline gain is throughput. vLLM reports V1 delivers up to 1.7x higher throughput than V0 on the same hardware without multi-step scheduling . The mechanism matters more than the number: the speedup comes from cutting Python/CPU overhead in the scheduler and orchestration loop, not from new GPU kernels. That has two practical consequences. First, there is no API change required — your existing serving code and request schema keep working. Second, the benefit scales with how CPU-bound your workload was, which is why the gains are not uniform.

Model-size variance is real. The speedup scales with how CPU-bound your workload was: smaller models (8B and below) see larger gains because they spend a higher fraction of wall-clock time in per-step CPU scheduling overhead, so removing that overhead helps them disproportionately. On larger models (70B+), the GPU is already the dominant cost and the CPU savings are proportionally smaller. Run a benchmark on your actual model and traffic mix rather than extrapolating from headline figures .

Beyond raw throughput, V1's operational advantages are the reason vLLM stays the default recommendation for general-purpose serving. Its cold start is about 62 seconds — second only to SGLang's ~58 seconds among the server-class engines, and vastly faster than TensorRT-LLM's per-model compilation step of roughly 28 minutes . Combined with the widest model coverage and the most complete documentation of the three, that makes vLLM the low-friction baseline: no compile wait, no narrow architecture support, and the smallest gap between "git clone" and a live endpoint.

The trade-off is that vLLM trails on specialized workloads. In raw high-concurrency throughput, SGLang posts ~1,920 tok/s versus vLLM's ~1,850 at 50 concurrent requests — a ~4% gap — and SGLang extends that lead further on shared-prefix traffic and lower single-request TTFT — the case the next section examines.

When SGLang Outperforms vLLM on Shared-Prefix Workloads

SGLang wins whenever requests share a common prefix — a system prompt, few-shot examples, or retrieved context. In the third-party H100 SXM5 benchmark (Spheron, Jan 2026, Llama-3.3-70B-Instruct FP8), SGLang reached about 1,920 output tok/s versus vLLM's ~1,850 at 50 concurrent requests — a ~4% gap on general mixed traffic . On prefix-heavy workloads, that gap widens through RadixAttention: SGLang's original benchmarks measured up to 5 times higher throughput when cache-hit rates are high (Jan 2024, A10G GPUs, vLLM v0.2.5 — directional signal; current H100 margins differ). The mechanism is RadixAttention.

RadixAttention stores KV-cache entries in a radix tree keyed by prompt-prefix hash. Two requests that begin with the same 2,000-token system prompt automatically reuse that cached prefix — no application code change, no manual cache key. This is exactly the structure of RAG pipelines, multi-turn chat, agent loops, and structured-JSON generation, where a large fixed context precedes a small variable tail. When the variable tail is short relative to the shared prefix, the cache hit rate climbs and SGLang's advantage widens.

The operational profile reinforces this. SGLang cold-starts in about 58 seconds, marginally ahead of vLLM's 62 seconds, and posts single-request TTFT in the 80–120 ms range — the lowest of the three major engines. That makes it the time-to-first-token leader without paying TensorRT-LLM's compile penalty, the trade-off the next section covers. In the same third-party H100 comparison at 50 concurrent requests, SGLang reached about 1,920 output tokens/s, behind TensorRT-LLM's ~2,100 but ahead of vLLM's ~1,850 — close enough that prefix reuse, not raw kernels, decides the outcome on prefix-heavy traffic.

The newer claims need a sharper eye. v0.5.8, released January 2026, integrates TensorRT-LLM's DeepSeek Sparse Attention (DSA) kernels and reports 3x–5x speedups on Blackwell, alongside an earlier February 2026 figure of up to 25x higher DeepSeek-R1 performance on GB300 NVL72 versus H200. These are project-reported and hardware-specific; they have not been validated at MLPerf scale, so treat them as directional rather than head-to-head. As one practitioner framing puts it:

"Rankings flip with workload — there is no single throughput winner; SGLang leads on shared-prefix traffic, vLLM on coverage, TensorRT-LLM on absolute peak," — benchmark analysis (source: Spheron, 2026).

The decision rule is concrete: if your traffic carries a large shared prefix and you can tolerate SGLang's narrower model coverage, the cache reuse pays for itself. If prefixes are short and unique, vLLM's broad-support profile holds.

TensorRT-LLM: The Throughput Lead vs. Its 28-Minute Compile Cost

TensorRT-LLM posts the highest absolute throughput of the major H100 engines, but it charges for that lead up front with a per-model compilation step. In the same third-party H100 SXM5 comparison (Llama-3.3-70B-Instruct FP8), TensorRT-LLM reached roughly 2,100 output tokens/s at 50 concurrent requests — ahead of SGLang's ~1,920 and vLLM's ~1,850 — while posting the lowest p50 time-to-first-token at 10 concurrent (105 ms vs. 112 ms and 120 ms) . The catch is cold start: where vLLM boots in ~62 s and SGLang in ~58 s, TensorRT-LLM spends roughly 28 minutes compiling a serving engine for each model before it answers a single request .

That compilation is the source of the speed, not an accident. TensorRT-LLM bakes a fixed model, precision, and batch profile into an optimized engine ahead of time, which is why controlled tests sometimes cite 2–4x gains over vLLM or TGI — and why every weight change, quantization swap, or LoRA variant means another compile pass . For a stable production model, you pay that 28 minutes once at deploy time and amortize it across weeks of traffic. For a team rotating through a model catalogue, the compile tax recurs on every rotation.

The standardized numbers confirm how far vendor-optimized TensorRT stacks scale. In MLPerf Inference Datacenter v5.1, NVIDIA's GB300 NVL72 (TensorRT 10.13 + CUDA 13.0) reported 420,659 tokens/s Offline and 209,328 tokens/s Server on DeepSeek-R1, and the GB200 NVL72 (TensorRT 10.11 + CUDA 12.9) reported 14,774.3 Offline and 11,614.3 Server on Llama 3.1 405B . Those are full rack-scale vendor submissions, not pure engine comparisons, so read them as ceilings rather than head-to-head engine rankings.

For capacity planning at a more attainable scale, the 8-GPU submissions are the useful anchor. Oracle's 8x B200 TensorRT entry reported 1,615.85 tokens/s Offline and 1,243.67 tokens/s Server on Llama 3.1 405B, and 145,789 Offline / 128,649 Server on the much smaller Llama 3.1 8B . The 405B figures give a concrete enterprise baseline: a single 8x B200 box serving a frontier-sized model lands in the low thousands of tokens/s under MLPerf's latency-constrained Server scenario, before any sharding across additional nodes.

The decision rule follows from the compile cost. TensorRT-LLM is the strongest fit for fixed-model production deployments — a single model held stable for weeks, where peak throughput and low TTFT directly cut serving cost and the one-time engine build is noise against the deployment lifetime. It is a poor fit for teams iterating on models, A/B testing checkpoints, or serving a dynamic catalogue, where the recurring ~28-minute compile and the rigid engine profile turn into operational drag. If your model set is small and frozen, TensorRT-LLM's absolute lead is yours to keep; if it churns, the boot-time engines stay more practical.

Local Inference: llama.cpp, Ollama, Apple Silicon, and the RTX 5090

On a single workstation the engine matters less than the configuration around it — memory residency and concurrency flags decide your throughput. Running Qwen 34B, Ollama (which sits on top of llama.cpp) reaches roughly 100 tokens/s, while llama.cpp's own llama-server hits about 124 tokens/s on a single request, climbs to around 231 tokens/s with --parallel and concurrency set to 128, and tops out near 826 tokens/s when you run multiple parallel instances (video: Alex Ziskind). The lesson is that the same weights span an 8x range depending entirely on how you batch them.

VRAM residency is the hard cliff. An RTX 5090 with 32 GB runs a 35B quantized MoE at 100–140 tokens/s as long as the weights stay fully GPU-resident; the moment they spill into system RAM, throughput collapses (video: Zen van Riel). There is no graceful degradation here — you are either inside the memory budget or off a cliff, so picking a quantization that leaves headroom matters more than chasing a faster kernel.

Apple's unified memory changes the math because the whole pool is addressable by the GPU. An M4 Pro exposes all 48 GB to the model, and a 512 GB Mac Studio can host four simultaneous Llama 70B servers (video: Alex Ziskind). On the engine side, vllm-mlx reports up to 525 tokens/s on an M4 Max — 21% to 87% higher than llama.cpp, a 4.3x aggregate advantage at 16 concurrent requests, and up to 28x faster on repeated images via content-based prefix caching . For Mac developers serving multiple agents, that concurrency multiplier is the deciding factor, not single-stream speed.

One configuration gotcha is worth flagging for anyone wiring a local server into Claude Code. LM Studio now exposes an Anthropic-compatible /v1/messages endpoint, but its default 4,000-token context silently hangs on Claude Code's roughly 80,000-token system prompt — the request never errors, it just stalls (video: Zen van Riel). Set context_length explicitly (about 80,000 is the recommended floor) before pointing ANTHROPIC_BASE_URL at the local server.

The practical takeaway for the consumer track: choose the engine for its API surface and concurrency support, then spend your tuning effort on quantization, memory residency, and the --parallel sweep. A tool like the "Llama Throughput Lab," which sweeps 308 instance and concurrency combinations per machine, exists precisely because the optimal setting is hardware-specific and not guessable from a spec sheet.

The 2026 Throughput Levers: FP4, KV Reuse, and Prefill-Decode Disaggregation

The biggest 2026 throughput gains no longer come from swapping engines — they come from four levers any of the major stacks can pull: FP4/NVFP4 quantization, FP8 KV cache, prefill-decode (PD) disaggregation, and prefix/KV reuse. FP4 is the single largest multiplier on Blackwell hardware. In vLLM's GB300 report (vLLM 0.14.1 + CUDA 13.0), DeepSeek-V3.2 running NVFP4 with TP2 hits 7,360 tokens/GPU/s in a prefill-only test, while DeepSeek-R1 in NVFP4+EP2 across two GB300s reaches 22,476 tokens/s prefill and 3,072 in an ISL=2k/OSL=1k mixed scenario . The same post reports Blackwell Ultra/B300 delivering roughly 8x prefill and 10–20x mixed-context gains over H200 in that configuration.

The lesson is that the precision format and parallelism layout now move numbers more than the engine name. That makes quantization and memory residency the first thing to tune, not the last.

PD disaggregation moved from research to production this year. Instead of one GPU pool handling both the compute-bound prefill and the latency-bound decode phase, disaggregation splits them so each scales independently. vLLM's April 2026 AMD MORI-IO article shows single-node PD disaggregation on one 8-GPU MI300X box running Qwen3-235B-A22B-FP8 at 8 req/s with 2,000-token prompts and 1,000-token outputs, achieving 2.5x higher SLO-compliant goodput than collocated serving . Goodput — requests served within your latency SLO, not raw tokens — is the metric that matters once you are routing real traffic.

Two research findings are worth keeping in view before you trust any vendor headline. SPEED-Bench warns that synthetic inputs overestimate real-world speculative-decoding throughput, so the accept rates you see in a benchmark rarely survive production prompt distributions. Separately, the Blink paper argues the host CPU remains a bottleneck across vLLM, SGLang, and TensorRT-LLM alike; by moving request orchestration off the CPU and onto the GPU/SmartNIC, it claims up to 8.47x lower P99 TTFT, 3.40x lower P99 TPOT, and 48.6% lower energy per token (figures from the paper; not an audited submission). Read both as a reminder that orchestration overhead and input realism can erase a paper's stated multiplier.

Model architecture can also override engine behavior outright. vLLM's GB300 data shows DeepSeek-R1 beating DeepSeek-V3.2 at 2k-token prefill, because V3.2's sparse-attention indexer path made a single DSA layer step run 2.7x the kernel time of MLA . V3.2 may still win at much longer contexts where sparsity pays off, but at short prefill the architecture, not the serving stack, decides the ranking.

"Blackwell Ultra/B300 showed up to 8x prefill and 10–20x mixed-context throughput over H200 in this setup, but the comparison between V3.2 and R1 flips with context length — the model's attention path dominates the result," — vLLM engineering team, GB300 DeepSeek report (source: vLLM, 2026-02).

The practical sequence: pick an engine that supports your model and precision, enable NVFP4 or FP8 where the hardware allows it, turn on KV/prefix reuse for repeated context, and reach for PD disaggregation once a single collocated pool can no longer hold your TTFT SLO. Then validate on your own traffic — synthetic wins do not transfer cleanly.

Choosing Your Engine: Decision Framework by Workload

Choose by your dominant workload, not by a headline throughput number. SGLang wins when requests share a prefix (RAG, multi-turn chat, structured JSON, agent loops), where RadixAttention KV-cache reuse delivers measurably higher throughput on prefix-heavy traffic — roughly 4% ahead on general H100 mixed-traffic tests, with the gap widening on prefix-dominant request patterns . vLLM V1 is the safe default for most teams. TensorRT-LLM posts the fastest absolute numbers for fixed-model production. Local dev belongs to llama.cpp, Ollama, or vllm-mlx.

Quick Answer: Match the engine to your workload: SGLang for shared-prefix traffic (RadixAttention KV-cache reuse; ~4% ahead on general H100 traffic, wider gap on prefix-dominant workloads), vLLM V1 as the broad-support default (within ~14% of TensorRT-LLM peak), TensorRT-LLM for static fixed-model serving despite its ~28-minute compile, and llama.cpp/Ollama or vllm-mlx for local hardware.

The decisive operational fact behind this split is cold start: vLLM warms in ~62 s and SGLang in ~58 s, while TensorRT-LLM pays a ~28-minute per-model compilation step (source). That cost is fine for a frozen catalogue and ruinous for a dynamic one. vLLM stays within a narrow margin of the leaders at 50 concurrent requests (~1,850 vs. TensorRT-LLM's ~2,100 output tok/s) (source) while carrying the widest model coverage and no compile step.

On local hardware the engine choice matters less than configuration. On Qwen 34B, llama.cpp's llama-server reached ~124 tok/s single-request and ~231 tok/s with --parallel concurrency=128, versus ~100 tok/s for Ollama (source). On Apple Silicon, vllm-mlx reports up to 525 tok/s on an M4 Max — 21%–87% above llama.cpp and 4.3x aggregate at 16 concurrent requests (source) — so reach for it when you need concurrent request scaling on a Mac (video: Zen van Riel).

The takeaway (결): there is no universal fastest engine in 2026. Default to vLLM V1, switch to SGLang the moment your traffic shares prefixes, and only accept TensorRT-LLM's compile tax when your model set is genuinely fixed. Then benchmark on your own request mix — the cross-source numbers here are directional, not head-to-head.

Watch / Sources

• Alex Ziskind — Your local LLM is 10x slower than it should be
• Zen van Riel — The Unbeatable Local AI Coding Workflow (Full 2026 Setup)
• Zen van Riel — The Ultimate Local AI Coding Guide For 2026

Frequently asked questions

Is vLLM still the right default inference engine for most teams in 2026?

Yes, for most teams. vLLM has the widest model coverage, no compilation step, and a roughly 62-second cold start , and its V1 engine removed the CPU overhead that capped V0 — reporting up to 1.7x higher throughput . It trails TensorRT-LLM by roughly 14% on peak H100 throughput (about 2,100 vs. 1,850 output tok/s at 50 concurrent requests) and trails SGLang on shared-prefix TTFT . Switch only when your own workload benchmarks show a meaningful gap — not on headline numbers.

How does SGLang's RadixAttention work, and which workloads benefit?

RadixAttention stores KV-cache entries keyed by a prompt-prefix hash, so requests that share a prefix — a system prompt, few-shot examples, or retrieval context — reuse the cache automatically with no code change. SGLang's 2024 benchmarks measured up to 5 times higher throughput when cache-hit rates are high (Jan 2024, A10G GPUs, vLLM v0.2.5 — directional signal; current H100 margins differ), while a Jan 2026 H100 third-party benchmark shows SGLang ~4% ahead on general mixed traffic . RAG, multi-turn chat, agent loops, and structured-JSON generation with stable templates gain the most. Fully unique prompts see no benefit, since there is no prefix to reuse.

Do MLPerf Inference results reflect real serving performance?

Directionally yes, exact numbers no. MLPerf's Offline scenario measures maximum batch throughput under ideal conditions, while Server measures Poisson-arrival throughput under latency constraints . Both use vendor-optimized full stacks rather than clean engine comparisons — for example AMD's 87x MI355X cluster reaching 1,042,110 tokens/s Offline on llama2-70b-99 . Real deployments add routing, autoscaling, and mixed traffic, so expect 20–40% lower effective throughput. MLCommons also allows post-submission modifications through change logs, so treat published figures as revisable.

What is prefill-decode disaggregation and when should I enable it?

Prefill-decode (PD) disaggregation splits prompt processing (prefill, compute-bound) and token generation (decode, memory-bandwidth-bound) onto distinct GPU allocations on one node or across nodes. Collocated serving runs both on the same GPUs and wastes each phase's strength. Enable it when serving long-context prompts (2k+ tokens) at meaningful request rates. vLLM's April 7, 2026 MI300X experiment with Qwen3-235B-A22B-FP8 (8 req/s, 2,000-token prompts, 1,000-token outputs) reported 2.5x higher SLO-compliant goodput than collocated serving on a single 8-GPU box .

Can I point Claude Code or Kilo Code at a local llama.cpp or Ollama server?

Yes, through the ANTHROPIC_BASE_URL and ANTHROPIC_API_KEY environment variables. LM Studio exposes an Anthropic-compatible /v1/messages endpoint and Ollama exposes an OpenAI-compatible /v1, so Claude Code, Kilo Code, and Continue can target a local server (video: Zen van Riel). The critical gotcha: LM Studio's default context window is 4,000 tokens, which silently hangs Claude Code's large system prompt — set context_length to roughly 80,000 in the model config before testing .