The most useful line in Charles Azam’s write-up isn’t that Fable 5 crushed an unpublished NP-hard optimization problem — it’s that the /goal mode “can win most runs and still be a bad default.” That gap between average win rate and worst-case behavior is where most agent evaluations quietly lie to you.

This maps straight onto anything with routing, retries, or goal-directed loops in production: a control-loop change has to be measured per-distribution and at the tail, never on aggregate pass rate. The HN discussion is worth skimming for the reproducibility notes.

So here’s the trap I’d watch for: how many teams have shipped a “reasoning” or “goal” toggle as a default because it won the A/B on mean score, without ever looking at what it does to the p99 run?