openrouter-caching-strategy

OpenRouter Caching Strategy

Overview

OpenRouter charges per token, so caching identical or similar requests can dramatically cut costs. Deterministic requests (temperature=0) with the same model and messages produce identical outputs -- these are safe to cache. This skill covers in-memory caching, persistent caching with TTL, and Anthropic prompt caching via OpenRouter.

Prerequisites

• An OpenRouter API key (sk-or-v1-...) exported as OPENROUTER_API_KEY — see the openrouter-install-auth skill for setup
• Python 3.8+ with the OpenAI SDK, plus the redis client package for the persistent cache; Node.js 18+ with the OpenAI SDK for the TypeScript variant in the references
• A Redis server reachable at localhost:6379 for Persistent Cache with Redis (the in-memory LLMCache needs no infrastructure)
• Deterministic request settings — caching is only safe at temperature=0

Instructions

1. Confirm the requests you want to cache are deterministic (temperature=0); non-zero temperatures produce different outputs each call and must never be cached.
2. Start with the In-Memory Cache: LLMCache plus cached_completion() gives you TTL expiry and hit/miss counters in a single process.
3. For multi-instance deployments, switch to Persistent Cache with Redis — redis_cached_completion() stores results under or:<sha256> keys with r.setex TTL expiry and falls through to a direct API call on a miss.
4. Build keys per Cache Key Design: include the model ID (with variants like :floor), messages, temperature, max_tokens, and top_p; exclude stream and the HTTP-Referer/X-Title headers.
5. For large static system prompts (RAG context), add cache_control: {"type": "ephemeral"} per Anthropic Prompt Caching via OpenRouter — cache reads bill at 0.1x the input rate.
6. Wire the Cache Invalidation table: flush per-model keys on model version updates, flush everything on system prompt changes, and let TTL handle the rest.

In-Memory Cache

import os, hashlib, json, time
from typing import Optional
from openai import OpenAI

client = OpenAI(
    base_url="https://openrouter.ai/api/v1",
    api_key=os.environ["OPENROUTER_API_KEY"],
    default_headers={"HTTP-Referer": "https://my-app.com", "X-Title": "my-app"},
)

class LLMCache:
    def __init__(self, ttl_seconds: int = 3600):
        self._cache: dict[str, tuple[dict, float]] = {}
        self._ttl = ttl_seconds
        self.hits = 0
        self.misses = 0

    def _key(self, model: str, messages: list, **kwargs) -> str:
        blob = json.dumps({"model": model, "messages": messages, **kwargs}, sort_keys=True)
        return hashlib.sha256(blob.encode()).hexdigest()

    def get(self, model: str, messages: list, **kwargs) -> Optional[dict]:
        k = self._key(model, messages, **kwargs)
        if k in self._cache:
            data, ts = self._cache[k]
            if time.time() - ts < self._ttl:
                self.hits += 1
                return data
            del self._cache[k]
        self.misses += 1
        return None

    def set(self, model: str, messages: list, response: dict, **kwargs):
        k = self._key(model, messages, **kwargs)
        self._cache[k] = (response, time.time())

cache = LLMCache(ttl_seconds=1800)

def cached_completion(messages, model="anthropic/claude-3.5-sonnet", **kwargs):
    """Only cache deterministic requests (temperature=0)."""
    kwargs.setdefault("temperature", 0)
    kwargs.setdefault("max_tokens", 1024)

    cached = cache.get(model, messages, **kwargs)
    if cached:
        return cached

    response = client.chat.completions.create(model=model, messages=messages, **kwargs)
    result = {
        "content": response.choices[0].message.content,
        "model": response.model,
        "usage": {"prompt": response.usage.prompt_tokens, "completion": response.usage.completion_tokens},
    }
    cache.set(model, messages, result, **kwargs)
    return result

Persistent Cache with Redis

import redis, json, hashlib

r = redis.Redis(host="localhost", port=6379, db=0)

def redis_cached_completion(messages, model="openai/gpt-4o-mini", ttl=3600, **kwargs):
    """Cache in Redis with automatic TTL expiry."""
    kwargs["temperature"] = 0  # Must be deterministic
    key = f"or:{hashlib.sha256(json.dumps({'m': model, 'msgs': messages, **kwargs}, sort_keys=True).encode()).hexdigest()}"

    cached = r.get(key)
    if cached:
        return json.loads(cached)

    response = client.chat.completions.create(model=model, messages=messages, **kwargs)
    result = {
        "content": response.choices[0].message.content,
        "model": response.model,
        "tokens": response.usage.prompt_tokens + response.usage.completion_tokens,
    }
    r.setex(key, ttl, json.dumps(result))
    return result

Anthropic Prompt Caching via OpenRouter

Anthropic models on OpenRouter support prompt caching -- large system prompts are cached server-side, reducing input cost by 90% on cache hits.

# Mark large static content blocks with cache_control
response = client.chat.completions.create(
    model="anthropic/claude-3.5-sonnet",
    messages=[
        {
            "role": "system",
            "content": [
                {
                    "type": "text",
                    "text": "You are an expert. Here is the full source:\n" + large_context,
                    "cache_control": {"type": "ephemeral"},  # Cache this block
                }
            ],
        },
        {"role": "user", "content": "What does the main() function do?"},
    ],
    max_tokens=1024,
)
# First call: cache_creation_input_tokens charged at 1.25x
# Subsequent: cache_read_input_tokens charged at 0.1x (90% savings)

Cache Key Design

def cache_key(model: str, messages: list, **params) -> str:
    """Deterministic cache key. Include everything that affects output.

    Include: model ID (with variant like :floor), messages, temperature,
    max_tokens, top_p, transforms, provider routing.
    Exclude: stream (doesn't affect content), HTTP-Referer, X-Title.
    """
    canonical = json.dumps({
        "model": model, "messages": messages,
        "temperature": params.get("temperature", 0),
        "max_tokens": params.get("max_tokens"),
        "top_p": params.get("top_p"),
    }, sort_keys=True)
    return hashlib.sha256(canonical.encode()).hexdigest()

Cache Invalidation

Output

• Cached completion payloads returned without an API round-trip: {"content", "model", "usage"} from the in-memory cache or {"content", "model", "tokens"} from Redis
• Redis keys of the form or:<sha256-of-canonical-request> that expire automatically via TTL
• Hit/miss counters and a hit_rate figure you can use to justify the caching infrastructure
• On Anthropic models, cache_creation_input_tokens billed at 1.25x on the first call and cache_read_input_tokens at 0.1x (90% savings) on subsequent hits

Examples

Two identical deterministic calls through the ResponseCache from the references — the second returns instantly from cache:

result1 = cached_completion("What is Python?")   # [Cache MISS] key=3f8a92c1... (stored)
result2 = cached_completion("What is Python?")   # [Cache HIT] key=3f8a92c1...
print(f"Hit rate: {cache.hit_rate:.0%}")         # Hit rate: 50%

More worked examples, including a TypeScript Redis-style cache: references/examples.md.

Error Handling

Enterprise Considerations

• Only cache deterministic requests (temperature=0) -- non-zero temperatures produce different outputs each time
• Use Anthropic prompt caching for large system prompts (RAG context) -- 90% cost reduction on cache hits
• Set TTL based on content freshness needs (30 min for dynamic, 24h for reference data)
• Track cache hit rate to justify caching infrastructure cost
• Use Redis or Memcached for multi-instance deployments; in-memory only works for single-process
• Version cache keys when updating system prompts or switching model versions

References

• Examples | Errors
• Prompt Caching | Models API