The move in CritICL is one of those “why weren’t we already doing this” ideas: a small model’s failure modes are structured and repeat predictably within a model family, so you can mine them once and feed them to a stronger model as critique-style in-context examples. Instead of paying for repeated sampling or an external verifier at inference time, you hand the strong model a profile of how its weaker sibling tends to be wrong — and let it route around those traps.

For anyone running an inference budget in production, the interesting part is the cost curve. Test-time scaling — best-of-N, self-consistency, verifier reranking — buys accuracy with tokens, and those tokens land on the P99 and the bill. CritICL reports competitive results with far fewer generations by front-loading the work: the static variant precomputes a global failure-mode profile, the dynamic variant retrieves input-specific critiques per query. That’s as much an eval-harness question as a modeling one — you only trust “failure modes are stable across scales” if you’ve measured it in your own family, not taken it on faith.

What I’d actually do with this: treat the failure-mode profile as a versioned artifact you regression-test, not a prompt you author once. The whole thing rests on weak-model failures predicting strong-model failures within a family — and the day that drifts, on a new checkpoint or a new domain, your critiques go stale and a wrong critique is a wrong steer applied to every token. Grounding it in a harness that catches that drift is the line between a cheap accuracy win and a silent regression.

The arXiv page has the full method; the HF paper page is where reaction will surface once people run it past math benchmarks. My question: does “failures are structured” survive on tool-use and retrieval-grounded tasks, where wrong is far fuzzier than a graded math answer?

tags: [ llm-ops ] [ research ]