What is the fastest way to serve an open-weight model with high throughput?

As of October 12, 2026: serve it with vLLM (v0.31.0, October 5, 2026) or SGLang (v0.5.21, October 2, 2026) on enough GPU memory to hold the weights plus a large KV cache, turn on continuous batching, prefix caching and an FP8 or FP4 quantized checkpoint, and set tensor or expert parallelism to fit the model. Use SGLang when many requests share long prefixes (agents, RAG with a fixed system prompt), vLLM as the general default, and TensorRT-LLM when you are standardised on NVIDIA and can spend engineering time on tuning.

vLLM vs SGLang vs TensorRT-LLM: which is fastest?

No engine wins every workload. SGLang's RadixAttention reuses KV cache across requests that share a prefix, which helps agent and RAG traffic most. vLLM has the broadest model and hardware support and day-one kernels for new open models. TensorRT-LLM compiles models for NVIDIA GPUs and can lead on raw per-GPU throughput, at the cost of more build and tuning work. Benchmark all three on your own prompt lengths before deciding.

Is Hugging Face TGI still a good choice for serving?

No for new deployments. Hugging Face put text-generation-inference into maintenance mode, accepting only minor fixes, and now recommends vLLM and SGLang for servers and llama.cpp or MLX for local use. Its last release, v3.3.7, shipped on December 19, 2025.

Should I self-host or use a hosted API for high throughput?

Use a hosted serverless API until your GPUs would stay busy. Providers such as Together AI and Fireworks AI charge per token for the same open models, so idle time costs nothing. Self-hosting wins when sustained utilisation is high, when you need a custom fine-tune at scale, or when data cannot leave your infrastructure.

Quick Answer

Fastest Way to Serve an Open-Weight Model at High Throughput

Published: October 12, 2026

The short answer

Throughput on an open-weight model is decided by how many requests fit on the GPU at once, so everything that frees GPU memory or reuses work raises it. As of October 12, 2026, the fastest path is:

Engine: vLLM (v0.31.0, October 5, 2026) as the default, SGLang (v0.5.21, October 2, 2026) for shared-prefix agent and RAG traffic, TensorRT-LLM (1.3.0 release candidates, latest rc29 on September 29, 2026) when you are all-NVIDIA and can invest in tuning.
Batching: continuous (in-flight) batching, which every modern engine does by default — never static batches.
Memory: paged KV cache plus a quantized checkpoint (FP8, or FP4 on Blackwell) so more sequences fit.
Reuse: prefix caching for repeated system prompts and documents.
Parallelism: tensor parallelism inside a node, expert parallelism for large MoE models, and data-parallel replicas behind a load balancer.

Do not start a new deployment on Hugging Face TGI: it is in maintenance mode and Hugging Face itself recommends vLLM and SGLang.

The engines compared

	vLLM	SGLang	TensorRT-LLM	llama.cpp
Latest release (Oct 12, 2026)	v0.31.0 (Oct 5)	v0.5.21 (Oct 2)	1.3.0rc29 (Sep 29)	Builds daily
Best at	Broad model + hardware support, general APIs	Shared-prefix workloads, structured output	Peak per-GPU throughput on NVIDIA	CPU, Apple Silicon, single user
KV-cache reuse	Automatic prefix caching	RadixAttention (tree-structured cache)	KV cache reuse	Prompt cache
Hardware	NVIDIA, AMD, TPU, others	NVIDIA, AMD, others	NVIDIA only	Almost anything
Effort	Low: `vllm serve <model>`	Low	High: engine build and tuning	Low
OpenAI-compatible server	Yes	Yes	Yes (via `trtllm-serve`)	Yes (`llama-server`)

Step by step

1. Size memory before anything else. Weights take roughly parameters × bytes per parameter (an FP8 checkpoint is about 1 byte per parameter). Whatever GPU memory is left becomes KV cache, and KV cache is your concurrency. A model that barely fits serves one or two users; leave 30–50% of memory for cache.

2. Pick the precision. FP8 roughly halves memory against BF16 with small quality loss on most current models; FP4 halves it again on GPUs that support it. Many 2026 open models ship official FP8 or FP4 checkpoints — prefer those to quantizing yourself, then run your own eval set.

3. Choose the engine by traffic shape.

Agents, RAG, chatbots with a long fixed system prompt: SGLang, because RadixAttention keeps shared prefixes in cache across requests.
Mixed public API traffic and many models: vLLM.
One model, very high volume, NVIDIA fleet, platform team available: TensorRT-LLM.

4. Set parallelism. Use tensor parallelism across the GPUs of one node (--tensor-parallel-size), expert parallelism for large mixture-of-experts models, and add replicas rather than stretching one instance across nodes when you can.

5. Tune the scheduler. Raise the maximum number of concurrent sequences until time-to-first-token approaches your limit; enable chunked prefill so long prompts do not stall decoding for everyone else; cap max_model_len at what you actually need, because a huge context limit reserves cache you rarely use.

6. Add speculative decoding for latency-bound traffic. A small draft model or multi-token-prediction head can raise tokens per second per user; at very high batch sizes the gain shrinks, so measure.

7. Benchmark on your own prompts. Replay real input and output lengths at 1×, 2× and 4× expected peak and record tokens per second, time to first token and inter-token latency. Published engine benchmarks rarely match your mix.

A minimal high-throughput launch

vllm serve <org>/<model>-FP8 \
  --tensor-parallel-size 4 \
  --enable-prefix-caching \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.92

The SGLang equivalent is python -m sglang.launch_server --model-path <org>/<model>-FP8 --tp 4.

When not to self-host

If your GPUs would sit idle most of the day, a per-token API is cheaper: Together AI and Fireworks AI serve the same open checkpoints and bill nothing when idle. Compare in leading serverless inference providers. For production hardening (gateway, autoscaling, evals) follow how to self-host an LLM for production; for desktop tools see vLLM vs Ollama vs LM Studio.

Last verified: October 12, 2026.

The short answer

The engines compared

Step by step

A minimal high-throughput launch

When not to self-host

Sources