Most agent evals grade task outputs. HarnessDev grades something we usually treat as fixed scaffolding: the execution harness itself — can the model build and then improve the infrastructure it runs inside?
- 🎯 The unit of evaluation moves from task outputs to runnable infrastructure — the agent starts from a minimal seed and a few cases, then builds a complete execution system.
- 📊 The scope is real: six creator LLMs, four domains, five downstream benchmarks, 2,207 held-out instances with the evaluation tasks hidden during development.
- ⚠️ Generated harnesses trail human-engineered ones on code and on search/research, but match or beat them on writing and ML experimentation — capability is domain-shaped, not uniform.
- 🔍 “Evolution” is fragile: iterative self-revision against execution feedback produces gains that are unstable and transfer only partially to held-out tasks.
- ⚡ The harness doesn’t port across models — the gains depend heavily on which model executes it, so a harness one model tuned can flop under another.
- 💡 The production takeaway: harness quality is a first-class variable, and any self-improving harness loop needs held-out validation and cross-model checks before you trust the numbers.
This matches what I see building agentic systems: model weights are rarely the bottleneck — routing, retries, tool wiring, and verification in the harness are. If an agent can measurably evolve its own harness but only for the model that grew it, we’re drifting toward per-model harnesses rather than one portable framework. The HF paper page has the early reactions. Would you ship a harness your agent tuned itself, knowing it only holds up under the exact model that wrote it?