The most useful finding in this study isn’t that skills help — we knew that. It’s why they help, and the answer reframes how I’d instrument a skill library in production.
The arXiv paper runs 8,135 controlled trials and lands on a distinction I’ve felt but never measured: skills work by procedural anchoring — turning noisy trajectories into stable action sequences — not by injecting missing facts. Procedural anchoring accounts for 65.7% of the cases where a skill helps; explicit knowledge injection, only 4.5%. If you’ve been writing SKILL.md files as knowledge dumps, that’s the wrong mental model. A skill earns its keep by stabilizing what the agent does next, not by teaching it something new.
The second finding should worry anyone running a large skill catalog: retrieval is a separate, brutal bottleneck. As the pool grows from 5 to 100 skills, actual-use precision collapses from 29.6% to 3.3%. The skill that would help is sitting in the library; the agent just can’t find it. And here’s the counterintuitive part — exact ground-truth invocation turns out to be neither necessary nor sufficient for task success. Confusable distractors wreck offline retrieval metrics while downstream success stays flat. So your skill-retrieval eval and your task-success eval can point in opposite directions, and grinding on the former can be wasted effort.
What I’d carry back to a real agent platform: stop scoring skills by aggregate task lift, which is where the HF paper page discussion keeps landing. Measure procedural stability and retrieval precision as separate axes, because they fail independently. A skill that anchors execution beautifully is worthless if a 100-skill pool buries it, and a perfectly retrieved skill still breaks under an incompatible context.
If procedural anchoring is two-thirds of the value, is the whole “skills as knowledge packages” framing backwards — should we be shipping skills as trajectory templates instead?