LLM queries are expensive and slow. Especially when 30–40% of them are duplicates or semantically similar. LLM caching is the cheapest way to reduce both. With 10 years of experience in AI/ML and over 50 caching implementation projects, we deliver turnkey systems: from simple exact match to semantic search via embeddings. Exact cache achieves up to 35% hit rate, Semantic cache adds another 28%, and together they cover 63% of queries without calling an LLM. At a typical load of 5000 queries/day, that's over $2,100/month savings. We'll assess your project in 1 day — just contact us.
The main pain point is every repeated request to GPT-4o or Claude hits the API and costs money. With 5000 queries/day, 30% are duplicates. Exact cache on Redis returns a response in 5–15 ms instead of 2–5 seconds — that's 100x faster. Semantic cache adds another 20–28% of matches by meaning, even with different wording. Context gaps from frequent similar questions are also eliminated — the cache guarantees a stable response without hallucinations.
How Exact Cache Works
Simple: we hash the prompt (messages, model, temperature) and store the answer in Redis. On a repeat request with the same hash, we return it without an LLM call. Suitable for FAQs, forms, template queries.
import hashlib
import json
import redis
from typing import Optional
from functools import wraps
class ExactLLMCache:
def __init__(self, redis_url: str = "redis://localhost:6379", ttl: int = 3600):
self.redis = redis.from_url(redis_url)
self.ttl = ttl
def _make_key(self, messages: list[dict], model: str, temperature: float) -> str:
"""Creates cache key from request parameters"""
cache_input = {
"messages": messages,
"model": model,
"temperature": temperature,
}
content = json.dumps(cache_input, sort_keys=True, ensure_ascii=False)
return f"llm:exact:{hashlib.sha256(content.encode()).hexdigest()}"
def get(self, messages: list[dict], model: str, temperature: float = 0) -> Optional[str]:
key = self._make_key(messages, model, temperature)
cached = self.redis.get(key)
if cached:
return cached.decode()
return None
def set(self, messages: list[dict], model: str, temperature: float, response: str):
key = self._make_key(messages, model, temperature)
self.redis.setex(key, self.ttl, response.encode())
def cached_complete(self, complete_fn):
"""Decorator for caching functions"""
@wraps(complete_fn)
def wrapper(messages, model="gpt-4o", temperature=0, **kwargs):
cached = self.get(messages, model, temperature)
if cached:
return cached
result = complete_fn(messages, model=model, temperature=temperature, **kwargs)
self.set(messages, model, temperature, result)
return result
return wrapper
Why Add Semantic Cache?
Exact cache only catches identical requests, but users often rephrase. Semantic Cache solves this: convert the question to an embedding, search a vector DB for similar ones, and if similarity >92%, return the answer. Effective for chats where wording varies.
from openai import OpenAI
import numpy as np
from dataclasses import dataclass
@dataclass
class CachedEntry:
query_embedding: list[float]
question: str
answer: str
model: str
created_at: float
class SemanticLLMCache:
"""Cache based on semantic question similarity"""
def __init__(
self,
similarity_threshold: float = 0.92,
max_entries: int = 10000,
):
self.openai = OpenAI()
self.threshold = similarity_threshold
self.entries: list[CachedEntry] = []
def _get_embedding(self, text: str) -> list[float]:
response = self.openai.embeddings.create(
model="text-embedding-3-small",
input=text,
)
return response.data[0].embedding
def _cosine_similarity(self, a: list[float], b: list[float]) -> float:
a_arr = np.array(a)
b_arr = np.array(b)
return np.dot(a_arr, b_arr) / (np.linalg.norm(a_arr) * np.linalg.norm(b_arr))
def get(self, question: str, model: str = None) -> Optional[str]:
"""Find similar question in cache"""
if not self.entries:
return None
query_embedding = self._get_embedding(question)
best_similarity = 0
best_answer = None
for entry in self.entries:
if model and entry.model != model:
continue
similarity = self._cosine_similarity(query_embedding, entry.query_embedding)
if similarity > best_similarity:
best_similarity = similarity
best_answer = entry.answer
if best_similarity >= self.threshold:
return best_answer
return None
def set(self, question: str, answer: str, model: str):
"""Add entry to cache"""
import time
embedding = self._get_embedding(question)
entry = CachedEntry(
query_embedding=embedding,
question=question,
answer=answer,
model=model,
created_at=time.time(),
)
self.entries.append(entry)
# Limit cache size
if len(self.entries) > 10000:
self.entries = sorted(self.entries, key=lambda e: e.created_at)[-10000:]
Comparison of Exact and Semantic Cache
| Characteristic | Exact Cache | Semantic Cache |
|---|---|---|
| Principle | Prompt hash | Vector similarity |
| Storage | Redis | ChromaDB / Qdrant |
| Latency | 5–15 ms | 50–100 ms |
| Hit rate | up to 35% | up to 28% |
| Use case | Frequent repeated requests | Semantically similar questions |
| Implementation complexity | Low | Medium |
Combined Cache: Redis + Vector Store
In production we combine both: first check exact cache (Redis), then semantic (ChromaDB). This gives minimal latency and maximum hit rate.
import chromadb
import time
class ProductionSemanticCache:
"""Production-ready cache: Redis for exact, Chroma for semantic"""
def __init__(self):
self.redis = redis.from_url("redis://localhost:6379")
self.chroma = chromadb.HttpClient(host="localhost", port=8000)
self.collection = self.chroma.get_or_create_collection("llm_cache")
self.openai = OpenAI()
self.similarity_threshold = 0.93
self.exact_ttl = 3600
self.semantic_ttl = 86400 # 24 hours
def get(self, question: str, model: str) -> Optional[dict]:
# 1. Exact match first (fast)
exact_key = f"llm:exact:{hashlib.md5(f'{question}:{model}'.encode()).hexdigest()}"
exact_hit = self.redis.get(exact_key)
if exact_hit:
return {"answer": exact_hit.decode(), "cache_type": "exact"}
# 2. Semantic match
embedding = self.openai.embeddings.create(
model="text-embedding-3-small",
input=question,
).data[0].embedding
results = self.collection.query(
query_embeddings=[embedding],
n_results=1,
where={"model": model},
)
if results["distances"] and results["distances"][0]:
distance = results["distances"][0][0]
similarity = 1 - distance # Chroma uses cosine distance
if similarity >= self.similarity_threshold:
answer = results["documents"][0][0]
return {"answer": answer, "cache_type": "semantic", "similarity": similarity}
return None
def set(self, question: str, answer: str, model: str):
# Exact cache in Redis
exact_key = f"llm:exact:{hashlib.md5(f'{question}:{model}'.encode()).hexdigest()}"
self.redis.setex(exact_key, self.exact_ttl, answer.encode())
# Semantic cache in Chroma
embedding = self.openai.embeddings.create(
model="text-embedding-3-small",
input=question,
).data[0].embedding
self.collection.add(
ids=[f"{int(time.time())}_{hash(question)}"],
embeddings=[embedding],
documents=[answer],
metadatas=[{"model": model, "question": question, "created_at": time.time()}],
)
How to Measure Cache Effectiveness?
We deploy metrics: hit rate (fraction of requests served from cache), p95 latency, cost per request. Typical results: exact hit 35%, semantic hit 28%, average latency drops from 2.3s to 0.4s. We use dashboards with alerts when hit rate falls below threshold. The 92% similarity threshold is empirically chosen to minimize false positives while preserving 95% of relevant matches. At threshold 0.95, hit rate drops 12%; at 0.90, incorrect responses increase.
Case Study: FAQ Bot with 5000 Queries/Day
One of our clients is a technical support service. Before caching, all requests went directly to GPT-4o. Results after configuring combined cache:
| Metric | Before Cache | After Cache |
|---|---|---|
| LLM cost | 100% | 37% |
| Average latency | 2.3s | 0.4s |
| Exact hit rate | 0% | 35% |
| Semantic hit rate | 0% | 28% |
Savings of 63% — just from caching, without changing the model. In monetary terms, that's over $2,100/month. Get an engineer consultation — we'll calculate savings for your project.
What Our Work Includes
- Analytics — audit your requests, identify patterns, estimate potential hit rate.
- Architecture — choose stack (Redis / Chroma / Qdrant), design cache schema.
- Implementation — write production code (Python, integrate with your LLM provider).
- Testing — A/B test with latency and cost measurements on your data.
- Monitoring — hit rate dashboard, latency dashboard, alerts on efficiency drops.
- Documentation and training — transfer code, provide instructions, train your team.
Estimated Timelines
- Exact cache (Redis): 0.5–1 day
- Semantic cache (Chroma + embeddings): 2–3 days
- Full production solution with monitoring: 1 week
We guarantee stable cache operation, post-implementation support, and transparent reporting. Order a project audit — we'll evaluate hit rate and savings in 1 day.