The interesting number in OreoLook’s three-layer caching paper isn’t about the model at all: an answer engine reported an 89.3% Redis keyspace hit rate with 0.1 ms reads on a single 8-vCPU CPU box, no GPU anywhere in the caching path. Synthesis still ships out to a remote provider. Everything expensive around it got solved with systems engineering most of us already know.

The three layers map cleanly onto where production RAG actually leaks time and money. A semantic query cache that catches rephrasings by cosine similarity on the embedding vector kills redundant LLM calls — the trick that pays for itself the moment two users ask the same thing three different ways. A URL embedding cache dedups embedding compute across sessions, the silent cost nobody budgets for until the embedding bill outruns the generation bill. And a rolling session context window in Redis, with LRU eviction to Huffman-compressed disk and re-hydration on demand, is what lets a conversation resume days later without pinning memory the whole time.

None of this is novel, and that’s the point. When teams chase RAG latency they reach for a faster reranker or a smaller model; the cheaper win is usually a cache keyed on the right thing — a normalized query embedding, a canonical URL — plus an eviction policy that treats idle sessions as cold storage instead of live state. 1.38 MB of overhead for that hit rate is the kind of figure that should embarrass a few over-provisioned setups.

The arXiv paper has the full deployment topology; the HF paper page carries the abstract and links. Would your RAG stack survive an audit of how many embeddings it recomputes per day?

tags: [ rag ] [ llm-ops ] [ research ]