Best LLM Latency Benchmarking Tools 2026: 5 Compared
The short answer
Use GuideLLM to find the request rate at which your endpoint still meets its latency target, NVIDIA AIPerf for the most detailed per-request metrics, and inference-perf for production-scale or Kubernetes tests. All three are Apache-2.0, speak the OpenAI-compatible API and report time to first token (TTFT) and inter-token latency (ITL) as distributions. Versions read from GitHub on November 18, 2026:
| Tool | Maintainer · version | Load patterns | Standout | Best for |
|---|---|---|---|---|
| GuideLLM | vLLM project · v0.8.1 (Oct 8, 2026) | synchronous, concurrent, throughput, constant, Poisson, sweep | Sweep finds max safe rate; HTML/CSV/JSON reports | Choosing a provider or GPU against an SLO |
| AIPerf | NVIDIA (ai-dynamo) · v0.13.0 (Sep 24, 2026) | concurrency, request rate, request count | TTFT, time to second token, ITL, per-user tok/s, p50–p99; MLflow/OTel/W&B export | Deep per-request analysis, NVIDIA stacks |
| inference-perf | Kubernetes wg-serving · v0.7.0 (Sep 15, 2026) | constant, Poisson, multi-stage, trace replay | Goodput (requests meeting SLO), 10K+ QPS, agentic and multi-turn workloads | Production capacity tests on vLLM, SGLang, TGI |
| vllm bench serve | vLLM · v0.34.0 (Nov 16, 2026) | request rate, concurrency | Built into vLLM | A quick check of your own vLLM server |
| k6 / Locust | Grafana (AGPL-3.0, v2.4.0) / Locust (MIT, 2.47.0) | any scripted scenario | General HTTP load testing | Testing the whole app, not just the model |
This is a spoke of our low-latency inference providers guide: measure here, then pick the provider there.
1. GuideLLM — best for SLO-driven provider choice
GuideLLM (pip install guidellm[recommended], Python 3.11–3.15, or the ghcr.io/vllm-project/guidellm container) simulates production-like traffic against any OpenAI-compatible or vLLM server and records full distributions for TTFT, ITL and end-to-end latency. Its sweep profile first measures synchronous and maximum throughput, then runs rates in between, so one command tells you the highest request rate at which p99 TTFT stays under your budget. Data can be synthetic (prompt_tokens=256,output_tokens=128), a Hugging Face dataset or local files, including multimodal. Results go to JSON, CSV, a self-contained HTML report or plots.
2. NVIDIA AIPerf — best for per-request detail
AIPerf replaces NVIDIA’s GenAI-Perf (the repository includes a migration guide) and runs with aiperf profile --model ... --streaming --url .... Its console table reports average, min, max and p50/p90/p99 for TTFT, time to second token, inter-token latency, request latency, and output tokens per second per user, which shows queueing that averages hide. Optional extras stream metrics to MLflow, OpenTelemetry or Weights & Biases, so latency runs can sit next to your training dashboards.
3. inference-perf — best for production scale
Built by the Kubernetes serving working group to standardize benchmarks, inference-perf measures TTFT, TPOT, ITL and normalized TPOT, plus goodput — the share of requests that met your latency SLO. Multi-stage runs step through rates to find saturation; trace replay re-runs real traffic (including OpenTelemetry traces and agentic tool loops); a multi-process load generator reaches 10,000+ QPS. It runs locally, in Docker or as a Kubernetes job and has verified support for vLLM, SGLang and TGI. inference-perf-convert translates vllm bench serve or aiperf profile arguments so results stay comparable.
4. vllm bench serve — quickest check
If you already run vLLM (v0.34.0 shipped November 16, 2026), vllm bench serve is the fastest way to get TTFT and throughput numbers without installing anything else. It is fine for comparing two configurations of the same server; for provider comparisons use GuideLLM or AIPerf so every endpoint is tested the same way.
5. k6 and Locust — test the whole application
Model benchmarks stop at the inference endpoint. If your latency budget includes retrieval, tool calls or your own API, script the full user journey in k6 (JavaScript, AGPL-3.0) or Locust (Python, MIT). Neither understands tokens, so record TTFT yourself from the stream.
How to run a fair latency comparison
- Fix the workload: same prompt and output token lengths, same model and quantization, streaming on.
- Measure percentiles at your expected rate, not just single-request latency; shared endpoints slow down under load.
- Turn reasoning down for interactive tests; hidden thinking tokens appear as TTFT.
- Run at peak hours for hosted APIs and repeat on two days.
- Report goodput against an explicit SLO (for example, p99 TTFT under 500 ms).
Once you have numbers, the fixes are in how to reduce LLM latency; for self-hosted servers, see lowest-latency LLM inference frameworks.
Last verified: November 18, 2026. Versions and release dates read from GitHub the same day.