Autonomous research agents get pitched as end-to-end: hypothesis to published paper, no human in the loop. This work’s real contribution is refusing to grade them on the final artifact and instead annotating exactly where the run breaks.
AutoResearchEval runs 100 tasks grounded in published frontier science across seven domains and the full lifecycle — ideation, retrieval, execution, analysis, writing, review — then evaluates eight harness-model combinations into 800 process-annotated trajectories. Out of that comes a 45-pattern failure taxonomy, attributed by a human-calibrated agent-as-a-judge that inspects intermediate artifacts, not just the endpoint. That’s the part I care about: process-level visibility instead of a single pass/fail number.
The finding that should bother anyone building agent frameworks is that all 45 patterns collapse into one deficit. Agents lack a metacognitive loop — the habit of checking what they produced against what they retrieved, revising when it doesn’t hold up, and questioning whether the path was even sound. And it recurs across all eight harness-model combinations, including the strongest models tested. The authors locate the gap at the model level, not the scaffold.
That’s the uncomfortable part. Most of us answer agent failures by adding orchestration: a verifier node, a retry policy, a critic pass. This says the same failure shows up no matter how you wire the graph, because the model itself doesn’t natively cross-check its own claims. Whether orchestration can paper over that is explicitly left open — they didn’t test it.
For my own eval harness the takeaway is concrete: treat “add a reflection step” as a hypothesis to measure, not a fix to ship. The HF paper page has the domain-level breakdown worth reading before you design your next agent’s self-correction path.
If the metacognitive gap really lives in the model, is your scaffolding fixing failures — or just relocating them?