Anyone who has shipped an agent knows the uncomfortable part: most of the reliability lives in the harness — the prompts, tool configs, and control logic wrapped around the model — not in the weights. AutoSaddler treats that harness as code and optimizes it offline from failure traces, and the framing is more useful than the benchmark deltas.
- 🔍 Diagnose, don’t reflect — it root-causes failure traces instead of asking the agent to “think about what went wrong,” which rarely survives a long-horizon task
- 🎯 Patch the harness as code — structured, targeted edits to prompts and tool wiring, not unconstrained rewrites that quietly regress everything else
- ✅ Validate before keeping — updates are selected on held-out mini-batches, so you retain generalizable fixes rather than trajectory-specific band-aids
- ⚡ Durable over one-shot — improvements persist across runs, which is the line between a demo and something you can operate
- 📊 The deltas are real — +9.0 on GAIA2, +9.6 on SWE-Bench Pro, +10.0 on Terminal-Bench 2.0 over the base harness
The part I keep circling back to: harness tuning today is manual, expensive, and mostly tribal knowledge — the thing a senior engineer does by staring at traces late at night. If “harness improvement as an offline learning problem” holds up outside these benchmarks, the shift that matters isn’t better scores; it’s that harness engineering stops being a craft and starts being a pipeline. The HF paper page has the ablations, and my bet is that diagnosis quality, not the patch generator, is what actually carries the gains.