The interesting claim here isn’t that agents are unreliable — we knew that. It’s that a single pass/fail number actively hides where a long-horizon run went sideways, and if you operate agents in production, that blind spot is the whole problem.
Beyond Final Scores evaluates seven frontier models across 36 long-horizon tasks, but the useful part is the framework: rule-based metrics that split a run into Solution Framing, Execution, and Feedback Control, plus controlled tests for whether an agent actually reuses experience across tasks. That decomposition is what turns “the eval dropped four points” into “the agent framed the problem fine but never corrected after bad feedback.”
The headline finding matches what I see running agentic workflows: today’s agents behave like engineering optimizers, not researchers. They combine known techniques competently, vary wildly run to run, and rarely produce genuine novelty. The variance is the operational killer — a harness that passes an eval once and fails the next run on the same inputs is untrustworthy no matter how high the mean.
What I’d take to my own eval harness: stop treating experience reuse as free. The paper shows accumulated context can mislead later decisions as often as it helps, which makes “just give the agent memory” a hypothesis to test, not a feature to ship. Same for scaffolding — in some tasks the harness design moved stability more than the model choice did.
If your agent eval is one number, you’re measuring whether it passed, not whether it’s stable enough to page you at 3am. The HF paper page has the task-level breakdown worth reading before your next eval refactor.
Would a process-level score have caught your last agent regression before it hit prod, or only explained it afterward?