Semantic Caching: Cut LLM Costs 50-80%

Semantic caching cuts LLM API costs by 40 to 80 percent on workloads where users ask similar questions in different wording. Instead of caching exact strings (which almost never match in natural language), semantic caching uses embeddings to recognize that “What’s the refund window?” and “How long do I have to return something?” are the same question and returns the cached answer. In 2026 this pattern moved from research-y to production baseline. This article walks through what semantic caching does, how to build one in 30 lines of Python, and the open-source and hosted options worth knowing.

What semantic caching actually is

Traditional cache (Redis GET/SET on exact string): “What is our refund policy?” hits cache. “What’s the refund policy?” misses because one character different. Cache hit rate in natural language chat: ~5-10%.

Semantic cache: embed every incoming question, compare to embeddings of cached questions, return the cached response if the similarity is above a threshold. “What is our refund policy?” and “What’s the refund policy?” both embed to nearly-identical vectors, both hit the same cache entry. Cache hit rate on real traffic: 40-70%.

The economic effect at 1M requests/month on GPT-4o:

• Fresh calls: 1M * $0.004 avg = $4,000
• With 50% semantic cache hit: 500K fresh + 500K cached = $2,000 + ~$50 embedding cost = $2,050
• Savings: $1,950/month, roughly $23K/year

Build-your-own in 30 lines

from openai import OpenAI
import numpy as np
from redis import Redis
import json, pickle

client = OpenAI()
redis = Redis()
SIMILARITY_THRESHOLD = 0.92

def embed(text):
    r = client.embeddings.create(model="text-embedding-3-small", input=text)
    return np.array(r.data[0].embedding)

def cosine(a, b):
    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))

def semantic_cache_get(query):
    query_vec = embed(query)
    # Fetch all cached entries (small cache; for large, use a vector DB)
    for key in redis.keys("cache:*"):
        entry = pickle.loads(redis.get(key))
        if cosine(query_vec, entry["vec"]) >= SIMILARITY_THRESHOLD:
            return entry["response"]
    return None

def semantic_cache_set(query, response):
    query_vec = embed(query)
    key = f"cache:{hash(query)}"
    redis.setex(key, 86400, pickle.dumps({"vec": query_vec, "response": response}))

def ask(query):
    cached = semantic_cache_get(query)
    if cached:
        return cached
    response = client.chat.completions.create(
        model="gpt-4o", messages=[{"role": "user", "content": query}]
    ).choices[0].message.content
    semantic_cache_set(query, response)
    return response

Works for small caches (under 10K entries). The linear scan over all keys becomes slow past that. For production scale, swap Redis for a vector database.

Production-scale: vector DB-backed cache

Any vector database can back a semantic cache. Common choices:

• Redis Stack: Redis with vector search module. Zero extra infrastructure if you already run Redis. Scales to 10M+ entries.
• Qdrant: open-source vector DB purpose-built for similarity search. Docker-deployable. Good filtering + metadata support.
• Pinecone: hosted vector DB. Zero ops, pay per usage. Simplest to start with.
• pgvector: PostgreSQL extension for vector search. Pick when you already run Postgres and want one less service.

Example with Qdrant:

from qdrant_client import QdrantClient
from qdrant_client.models import Distance, VectorParams, PointStruct

qdrant = QdrantClient("localhost", port=6333)
qdrant.create_collection(
    collection_name="llm_cache",
    vectors_config=VectorParams(size=1536, distance=Distance.COSINE),
)

def cache_get(query, threshold=0.92):
    vec = embed(query)
    hits = qdrant.search(collection_name="llm_cache", query_vector=vec.tolist(), limit=1)
    if hits and hits[0].score >= threshold:
        return hits[0].payload["response"]
    return None

def cache_set(query, response):
    vec = embed(query)
    qdrant.upsert(
        collection_name="llm_cache",
        points=[PointStruct(id=hash(query), vector=vec.tolist(), payload={"response": response})],
    )

Qdrant scales horizontally, supports filters (per-user cache, per-tenant cache), and runs on commodity hardware.

GPTCache: the open-source turnkey option

GPTCache (by Zilliz) is the most popular off-the-shelf semantic caching library. Drop-in integration with OpenAI, Anthropic, and LangChain.

pip install gptcache

from gptcache import cache
from gptcache.adapter import openai  # wraps openai module
from gptcache.embedding import OpenAI
from gptcache.manager import CacheBase, VectorBase, get_data_manager
from gptcache.similarity_evaluation.distance import SearchDistanceEvaluation

cache.init(
    embedding_func=OpenAI().to_embeddings,
    data_manager=get_data_manager(
        CacheBase("sqlite"),
        VectorBase("faiss", dimension=1536),
    ),
    similarity_evaluation=SearchDistanceEvaluation(),
)

# Now openai.ChatCompletion.create goes through the cache
response = openai.ChatCompletion.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "What is our refund policy?"}],
)

Zero code changes to your business logic. Caching happens inside the openai module. Supports Faiss, Milvus, Qdrant, Redis, pgvector as the vector backend.

Hosted options: Helicone and Portkey

Both are LLM gateways that offer semantic caching as a built-in feature. Switch your base URL to the gateway, caching happens automatically:

from openai import OpenAI

# Helicone
client = OpenAI(
    base_url="https://oai.helicone.ai/v1",
    default_headers={
        "Helicone-Auth": f"Bearer {HELICONE_KEY}",
        "Helicone-Cache-Enabled": "true",
        "Helicone-Cache-Semantic-Threshold": "0.92",
    },
)

Trade-off: hosted gateways add 50-100ms latency (gateway hop) but zero infrastructure to manage. Pricing is per cached request, which scales linearly with your traffic. For moderate volumes (under 1M requests/month) hosted options are the pragmatic pick; past that, self-hosted saves money.

The threshold tuning problem

Threshold too low (e.g., 0.80): false positive cache hits. “What is our refund policy?” matches “What is our shipping policy?” because both are policy questions. Users get wrong answers.

Threshold too high (e.g., 0.98): hit rate drops to near zero. Every minor phrasing difference misses.

Sweet spot: 0.90-0.94 for most workloads. Measure on your actual query distribution:

1. Collect 1,000 real production queries
2. For each query, find the nearest cached entry and record the similarity
3. Manually judge: at what similarity did matches start being incorrect?
4. Set threshold slightly above that line

Teams with user-facing chat typically land at 0.92. Teams with structured tasks (classification, extraction) can go lower (0.88) because tasks are more deterministic.

When semantic caching fails

• Highly personalized responses: if the answer depends on the specific user (balance, order history), caching across users produces leakage. Namespace the cache per user.
• Time-sensitive content: today’s price, latest news. Cache TTL must match the content’s freshness window (short TTL = less benefit).
• Multi-turn conversations: context from prior turns changes answers. Cache keys need to include relevant conversation state, not just the latest message.
• High-stakes decisions: medical advice, legal interpretation. Even a 1% wrong-match rate is unacceptable. Disable caching on these flows or use exact-match only.

Measuring your actual savings

Log every request with:

• Cache hit / miss
• Similarity score (for hits)
• Response was correct per user feedback (thumbs up/down)
• Cost that would have been incurred without cache

Dashboard shows real hit rate and real savings per week. Also catches false-positive cache hits when thumbs-down rate on cached responses exceeds thumbs-down rate on fresh responses. If you see this, raise the threshold.

Frequently Asked Questions

Joken E. Villanueva

Founder & Lead Developer at PIES IT Solution

Founder of PIES Information Technology Solutions, a software company building production-grade applications for institutions across the Philippines. Over 8 years of hands-on full-stack development experience, currently leading the development of ClinicAI, an AI-powered clinic management platform.

Expertise: PHP · MySQL · JavaScript · AI Integration · SaaS Architecture · VB.NET · Database Design · Capstone Documentation · Java  · View all posts by Joken E. Villanueva →