The easy story about Kimi K3 is the size — 2.8T parameters, the biggest open-weight model out right now, scaled up from last year’s 48B Kimi Linear. The more useful story, laid out in Sebastian Raschka’s architecture notes, is that almost every component swap is an inference-efficiency tweak. MoE becomes LatentMoE, standard attention becomes multi-head latent attention plus Kimi Delta Attention. The frontier trend isn’t “add capability,” it’s “serve the same capability for fewer FLOPs.”

That framing matters if you operate models rather than just benchmark them. When a lab pours its architecture budget into cheaper inference, it’s telling you where the real constraint lives — the serving bill and the latency budget, not the leaderboard. K3, DeepSeek V4, and Nemotron 3 are all converging on the same answer, which makes this a direction, not a one-off.

Two choices stand out. Attention residuals are the one change that isn’t about efficiency — they connect residuals across layers with an attention-weighted contribution score, costing ~4% in training and ~2% in inference for a consistent validation-loss win. That’s a deliberate spend, and worth watching whether it generalizes. The bolder bet is positional encoding: K3 drops RoPE entirely and runs NoPE everywhere. The recent orthodoxy was RoPE in local/sliding-window layers and NoPE in the global ones. K3 is the first frontier-scale model I’ve seen go all-NoPE, and it inherited that from Kimi Linear rather than bolting it on.

Native multimodal support rounds it out, but the architecture bet is the headline for anyone thinking about cost-per-token at scale. The HN discussion is circling the size number; the interesting question is downstream. If all-NoPE holds at 2.8T, how long before positional embeddings look like a workaround we kept out of habit?