Your Inference Bill Is Really a Memory Bill

Your Inference Bill Is Really a Memory Bill

When I plan a latency budget for a RAG or agent system, the conversation usually starts with model size and FLOPs. The Epoch AI breakdown of chip component costs is a good reminder that the economics have already moved somewhere else: memory.

Their numbers: HBM went from 52% of AI accelerator component cost in Q1 2024 to 63% by Q4 2025, with HBM spend climbing from roughly $12B to $32B in a year. Logic dies held flat around 13%. So the expensive part of the chip you’re renting is increasingly the memory stack, not the compute.

That tracks with what shows up in production serving:

🎯 KV cache is the real tenant — long-context agents and multi-turn RAG live or die on memory bandwidth, not raw matmul throughput.
⚡ Batching trades memory for latency — the knobs that improve P99 (bigger batches, longer context) are the same ones that pressure HBM.
📊 Cost-per-token is a memory curve — when hyperscalers bake in higher component prices (Microsoft cited $25B of it in FY2026 capex, Meta widened its range by $10B), that flows straight into what you pay per million tokens.

The practical takeaway for anyone running LLM ops: context length isn’t free, and “just raise the context window” is a memory-budget decision, not a prompt-engineering one. The teams that measure tokens-in-flight per request are going to forecast cost better than the teams measuring model params.

The HN thread has a good back-and-forth on whether HBM supply ever loosens. My bet: it doesn’t before 2027, and “context efficiency” quietly becomes a line item we actually optimize. What’s your current cost driver — tokens, or the memory holding them?