An agent that completes a stateful business task once out of twenty tries isn’t a working agent — it’s a demo that quietly corrupts state the other nineteen times. Thinkingbox, in this arXiv paper, makes that difference measurable: 507 policy-conditioned workflows across retail, hospitality, auto insurance, neobank IT, and consulting HR support, each graded on the terminal backend state rather than the model’s reply.

The headline is the gap between two numbers. The strongest model hits 65.36% pass@1 but only 25.25% pass^20 — passing once versus passing all twenty runs. In production that second number is the only one that matters; nobody ships a workflow that silently fails three times in four.

The detail that should worry anyone running an eval harness: many failed trials terminated cleanly, with valid, state-changing tool calls. Response-level and tool-call-level signals looked green while the end state was wrong. If your harness grades “did it call the tool” or “did it produce a plausible answer,” you’re scoring the wrong thing — outcome checks over persistent state are the only signal that survives contact with real workflows. The isolated MCP-compatible sandbox and full execution traces linked from the HF paper page are the reusable part here, not the leaderboard.

Early coverage lands on both sides of the same numbers. Pebblous reads the pass^20 collapse as constructive — a nudge to verify agents with repeated runs and backend-state checks instead of response logs — while Crypto Briefing frames the identical 65%-to-25% drop as evidence that even the best agents fail three times in four. The split is about framing, not findings: everyone now agrees the reliability gap is the story, and that shift in what we measure is overdue.

tags: [ agentic-ai ] [ llm-ops ] [ enterprise-ai ] [ research ]