Most context compression still round-trips through something human-readable — a text summary, or an OCR’d image — even when the only consumer downstream is a language model. That round-trip is wasted work, and in a production RAG pipeline it’s wasted latency.

LatentPress writes conversational history and long documents straight into continuous memory tokens that a frozen decoder reads through its input-embedding interface, with no text reconstruction at inference. The writer is small — an adapter of 4-26M parameters, roughly 0.1% of the decoder — so you aren’t paying to retrain the model that reads the compressed context. It compresses 4-16x, and on LongMemEval it reaches 0.504 accuracy at 7.7x compression against 0.490 for the uncompressed evidence, while text summaries collapse to 0.184 and OCR-based compression lands between 0.312 and 0.426.

The numbers I’d actually stress-test in production are the latency ones. Writing a conversation takes about 43ms, and reading the compressed memory runs 5-9x faster than reading raw context. For a system where long-context reads and reranking dominate the P99 budget, that read-side speedup is the interesting lever — not the accuracy headline. The caveat is stated honestly: at 16x compression it trails raw context, so this isn’t free above a point, and the writer is reader-matched, which couples you to a decoder. The HF paper page has the full ablations.

My bet: the teams who gain most here aren’t the ones chasing a maximum compression ratio, but the ones whose context is 90% stale conversation history they re-encode from scratch every single turn. Is your memory really a retrieval problem, or an encoding problem you’ve been solving in the wrong representation?

tags: [ rag ] [ llm-ops ] [ research ]