What is the best semantic caching tool for LLM apps in 2026?

As of November 12, 2026: LiteLLM's proxy cache if you already route traffic through LiteLLM (semantic backends Redis, Valkey and Qdrant, open source); RedisVL's SemanticCache (MIT, v0.32.0 released November 11, 2026) if you want a library in your own code on Redis; Redis LangCache for a managed REST service, though it is still a public preview with no published price; and Portkey if you want a hosted gateway, where semantic caching starts on the $49-a-month Production plan. GPTCache has not shipped a release since August 2024.

Is semantic caching safe for AI agents?

Usually not. LiteLLM's documentation warns that semantic caching is designed for single-shot prompts and will replay stale responses on multi-turn or agentic traffic, because two agent turns can look similar while depending on different tool results. Use exact-match caching and provider prompt caching for agents, and keep semantic caching for repeated standalone questions such as FAQ bots and search suggestions.

What is the difference between semantic caching and prompt caching?

Prompt caching is done by the model provider: a repeated prefix of your prompt is billed at a discount (for example Claude Opus 5.5 cache reads cost $0.20 per million tokens against $4 for fresh input), and the model still generates a fresh answer. Semantic caching happens in your stack: it embeds the incoming prompt, finds a previous prompt above a similarity threshold, and returns the stored answer without calling the model at all, saving the output tokens too.

How much does semantic caching save?

Roughly your output-token spend times your cache hit rate, minus the cost of embeddings and the cache store. Redis's own formula for LangCache is monthly output token cost multiplied by cache hit rate: a $200 monthly LLM bill with 60% of spend on output and a 50% hit rate saves about $60. Hit rates depend entirely on how repetitive your traffic is, so measure them on logged prompts before buying.

Quick Answer

Best Semantic Caching Tools for LLM Apps in 2026

Published: November 12, 2026

The short answer

Use the cache that lives where your traffic already flows. If you run LiteLLM, turn on its semantic cache; if you write Python against Redis, use RedisVL; if you want someone else to run it, Redis LangCache or Portkey. As of November 12, 2026:

Tool	Type	Backends	Price	Status	Best for
LiteLLM proxy cache	Gateway feature (open source)	Redis, Valkey, Qdrant (semantic); Redis, S3, GCS, memory, disk (exact)	Free; you pay for the store and embeddings	Active (v1.109.1, Nov 12, 2026)	Teams already behind a LiteLLM gateway
RedisVL SemanticCache	Python library	Redis	Free (MIT)	Active (v0.32.0, Nov 11, 2026)	Caching inside your own app code
Redis LangCache	Managed REST API on Redis Cloud	Redis Cloud; self-managed on Kubernetes	No published price (public preview)	Preview	A hosted service with an embedding model built in
Portkey	Hosted AI gateway	Managed	Simple + semantic caching on Production, $49/month (100k logs)	Active	Teams that want caching with logs, guardrails and routing
GPTCache	Python library	Many	Free (MIT)	Last release 0.1.44, Aug 1, 2024	Legacy projects only

Cloudflare AI Gateway also caches, but only identical requests; its docs say semantic matching is planned, not shipped.

When semantic caching pays off — and when it hurts

Semantic caching returns a stored answer when a new prompt is similar enough to an old one. That is great for:

FAQ and support bots where thousands of users ask the same 200 questions,
search suggestions and classification of short, repeated inputs,
internal tools where the same report question recurs daily.

It is risky for anything where the right answer depends on context the embedding cannot see: multi-turn chat, agents, personalised answers, or data that changes. LiteLLM’s docs put it bluntly — semantic caching “is designed for single-shot prompts” and on agentic traffic “will replay stale responses”. For agents, rely on provider prompt caching and exact-match caching instead.

Tool notes

LiteLLM. Set cache: True with type: redis-semantic, valkey-semantic or qdrant-semantic and a similarity_threshold (the docs’ example uses 0.8). Responses from the semantic cache carry an x-litellm-semantic-similarity header, so you can log how close each hit was. LiteLLM’s docs also warn that an in-memory cache is per worker: four workers mean four caches and roughly a quarter of the hit rate, so use Redis past one worker. More gateways in best LLM gateway.

RedisVL. from redisvl.extensions.cache.llm import SemanticCache, set a name, TTL and distance threshold, and call check() before your model and store() after. It is the lowest-friction option if Redis is already in your stack, and it is actively released.

Redis LangCache. A REST API on Redis Cloud that handles embeddings and storage for you, with a self-managed Kubernetes option. Redis’s docs still label it a public preview and publish no price; budget for a change when it reaches GA.

Portkey. Its free Developer tier includes only simple (exact) caching; semantic caching starts on Production at $49 a month with 100,000 recorded logs, then $9 per extra 100,000 requests.

GPTCache. Once the default choice, but its last release was August 2024. Prefer one of the maintained options for new work.

How to set it up safely

Log a week of prompts first and measure how many near-duplicates exist; that is your ceiling hit rate.
Start strict (high similarity threshold), then loosen while spot-checking hits.
Partition the cache by user, tenant, model and system prompt, so one customer never receives another’s answer.
Set a TTL that matches how fast your facts change.
Exclude tool-using and multi-turn requests.

The full cost playbook — caching, batching and routing together — is in how to cut AI API costs; model rates are on our current API prices table.

Last verified: November 12, 2026.

The short answer

When semantic caching pays off — and when it hurts

Tool notes

How to set it up safely

Sources