AI agents · OpenClaw · self-hosting · automation

Quick Answer

Best LLM Latency Benchmarking Tools 2026: 5 Compared

Published:

The short answer

Use GuideLLM to find the request rate at which your endpoint still meets its latency target, NVIDIA AIPerf for the most detailed per-request metrics, and inference-perf for production-scale or Kubernetes tests. All three are Apache-2.0, speak the OpenAI-compatible API and report time to first token (TTFT) and inter-token latency (ITL) as distributions. Versions read from GitHub on November 18, 2026:

ToolMaintainer · versionLoad patternsStandoutBest for
GuideLLMvLLM project · v0.8.1 (Oct 8, 2026)synchronous, concurrent, throughput, constant, Poisson, sweepSweep finds max safe rate; HTML/CSV/JSON reportsChoosing a provider or GPU against an SLO
AIPerfNVIDIA (ai-dynamo) · v0.13.0 (Sep 24, 2026)concurrency, request rate, request countTTFT, time to second token, ITL, per-user tok/s, p50–p99; MLflow/OTel/W&B exportDeep per-request analysis, NVIDIA stacks
inference-perfKubernetes wg-serving · v0.7.0 (Sep 15, 2026)constant, Poisson, multi-stage, trace replayGoodput (requests meeting SLO), 10K+ QPS, agentic and multi-turn workloadsProduction capacity tests on vLLM, SGLang, TGI
vllm bench servevLLM · v0.34.0 (Nov 16, 2026)request rate, concurrencyBuilt into vLLMA quick check of your own vLLM server
k6 / LocustGrafana (AGPL-3.0, v2.4.0) / Locust (MIT, 2.47.0)any scripted scenarioGeneral HTTP load testingTesting the whole app, not just the model

This is a spoke of our low-latency inference providers guide: measure here, then pick the provider there.

1. GuideLLM — best for SLO-driven provider choice

GuideLLM (pip install guidellm[recommended], Python 3.11–3.15, or the ghcr.io/vllm-project/guidellm container) simulates production-like traffic against any OpenAI-compatible or vLLM server and records full distributions for TTFT, ITL and end-to-end latency. Its sweep profile first measures synchronous and maximum throughput, then runs rates in between, so one command tells you the highest request rate at which p99 TTFT stays under your budget. Data can be synthetic (prompt_tokens=256,output_tokens=128), a Hugging Face dataset or local files, including multimodal. Results go to JSON, CSV, a self-contained HTML report or plots.

2. NVIDIA AIPerf — best for per-request detail

AIPerf replaces NVIDIA’s GenAI-Perf (the repository includes a migration guide) and runs with aiperf profile --model ... --streaming --url .... Its console table reports average, min, max and p50/p90/p99 for TTFT, time to second token, inter-token latency, request latency, and output tokens per second per user, which shows queueing that averages hide. Optional extras stream metrics to MLflow, OpenTelemetry or Weights & Biases, so latency runs can sit next to your training dashboards.

3. inference-perf — best for production scale

Built by the Kubernetes serving working group to standardize benchmarks, inference-perf measures TTFT, TPOT, ITL and normalized TPOT, plus goodput — the share of requests that met your latency SLO. Multi-stage runs step through rates to find saturation; trace replay re-runs real traffic (including OpenTelemetry traces and agentic tool loops); a multi-process load generator reaches 10,000+ QPS. It runs locally, in Docker or as a Kubernetes job and has verified support for vLLM, SGLang and TGI. inference-perf-convert translates vllm bench serve or aiperf profile arguments so results stay comparable.

4. vllm bench serve — quickest check

If you already run vLLM (v0.34.0 shipped November 16, 2026), vllm bench serve is the fastest way to get TTFT and throughput numbers without installing anything else. It is fine for comparing two configurations of the same server; for provider comparisons use GuideLLM or AIPerf so every endpoint is tested the same way.

5. k6 and Locust — test the whole application

Model benchmarks stop at the inference endpoint. If your latency budget includes retrieval, tool calls or your own API, script the full user journey in k6 (JavaScript, AGPL-3.0) or Locust (Python, MIT). Neither understands tokens, so record TTFT yourself from the stream.

How to run a fair latency comparison

  1. Fix the workload: same prompt and output token lengths, same model and quantization, streaming on.
  2. Measure percentiles at your expected rate, not just single-request latency; shared endpoints slow down under load.
  3. Turn reasoning down for interactive tests; hidden thinking tokens appear as TTFT.
  4. Run at peak hours for hosted APIs and repeat on two days.
  5. Report goodput against an explicit SLO (for example, p99 TTFT under 500 ms).

Once you have numbers, the fixes are in how to reduce LLM latency; for self-hosted servers, see lowest-latency LLM inference frameworks.

Last verified: November 18, 2026. Versions and release dates read from GitHub the same day.

Sources