The number I can’t stop thinking about in AgentJudgeBench isn’t the benchmark size — it’s 77 to 82%. That’s the narrow band every judge they tested, from 20B up to frontier scale, collapses into on hard tool-calling workflows when there’s no ground truth to check against. Scaling the judge doesn’t move it. That’s a structural ceiling, and if you gate agentic systems on an LLM judge, it’s your ceiling too.
Most of us reach for LLM-as-a-judge exactly where ground truth is expensive: multi-step tool-calling over a workflow DAG, where “did the agent do the right thing” has no cheap oracle. The paper’s uncomfortable result is that this is precisely the regime where judges are least reliable — alignment degrades monotonically with difficulty, and 1.5x faster once you remove the ground truth. The mitigations we habitually reach for barely register: chain-of-thought and judge temperature are noise, while structured rubrics buy up to 6.5 points but don’t generalize across judge-generator pairs. Even ground truth isn’t a free win — it made GPT-5.4 and Gemini-2.5-Pro worse, apparently from over-anchoring on the reference.
I’ve built enough eval harnesses to have quietly assumed a bigger judge model would eventually paper over rubric gaps. This says it won’t. The practical read: treat the judge as a measurement instrument with a known error floor, budget for that floor in your SLAs, and spend your effort on rubric design and ground-truth coverage for the hard tiers rather than on a more expensive judge. The HF paper page has the full breakdown by DAG topology.
If your production agent evals run judge-only on the hard cases, what’s your actual confidence interval — and have you ever measured it against a human panel?