Fine-tuning a model to just remember a stream of facts sounds simple until you watch retention collapse. The long-horizon memorization setting in this paper — 100 query-answer tasks learned by sequential fine-tuning, no replay, no task IDs at inference — drives naive supervised fine-tuning down to 1.2% final retention. The headline isn’t a new mechanism; it’s that no single mechanism survives the horizon, and composing the right ones does.
- 🎯 Composition beats any single fix: stacking data, function, and weight anchors with merged LoRA lifts average final retention from 1.2% to 34.9% — a 28x gain over naive sequential fine-tuning.
- 🔍 Two design axes, not one: the anchors decide what prior information each update preserves; the low-rank allocation rule decides where successive updates live. Separating those is the actual contribution.
- ⚡ Data anchor + merged LoRA interact super-additively across all three datasets — the pair beats the sum of its parts, which is rare enough to design around.
- 📊 The interactions are measured, not asserted: task-level successive halving searches the combinatorial space and a factorial experiment isolates individual and interaction effects.
- ⚠️ 34.9% is not “solved”: two-thirds of what the model learned is gone by task 100. This narrows catastrophic forgetting; it doesn’t close it.
- 💡 The production read: this is the fine-tune-to-internalize alternative to retrieval, and it says internalized memory is a systems problem — anchors plus allocation — not a single-hyperparameter tweak.
The HF paper page frames this as memorization, but the real target is any model you keep updating in place instead of re-indexing. If you’re choosing between fine-tuning knowledge in and retrieving it at query time, does 34.9% retention change the math for your slice of facts?
tags: [ research ] [ llm-ops ] [ rag ]