The most useful idea in Cognition’s FrontierCode isn’t the leaderboard — it’s the target metric. They grade on whether a maintainer would actually merge the PR, not just whether the code passes. Once AI writes most of the diff, “correct” stops being the bar and “mergeable” starts being it.

That shift matches what eval harnesses for agentic coding run into in production: a green test suite tells you almost nothing about whether a change respects scope, style, and the unwritten conventions of a codebase.

• 🎯 Mergeability over correctness — grading the thing maintainers care about, not the thing that’s easy to measure
• 🔍 Ensemble grading — unit tests plus rubrics plus new verifiers, because no single signal captures “good code”
• ⚠️ Rubrics are subjective — they lean on adversarial QC and multi-stage human review to keep the judge honest, the same fight every LLM-as-judge harness has
• 📊 81% lower false-positive rate than SWE-Bench Pro — the real headline is trusting the grader, not the model
• 💡 Tasks from real maintainers — 40 hours per task from people who own the repos, which is where the realism comes from

The honest result: even frontier models struggle on it. That’s the point — a benchmark that’s already saturated teaches you nothing, and most coding benchmarks are nearly there.

The HN discussion splits on whether subjective rubric grading can ever be reproducible. I think that’s the wrong worry. The harder question is whether “mergeable” generalizes across codebases, or whether every team ends up needing its own verifier set. If quality is local, can any single benchmark stay meaningful?