The number that matters in this experiment isn’t the $1,500 spend — it’s the distance between “the model spots a vulnerability” and “the model chains it into a working exploit without a human nudging it back on track.” Stand up a deliberately vulnerable app, point frontier models at it, and that gap is exactly what you’re measuring.

Single-shot accuracy is the wrong metric here, and it’s the wrong metric for most agentic work I deal with. A model that recognizes a SQL injection in isolation tells you almost nothing about whether an agent can plan a multi-step attack, recover when a payload fails, and verify it actually got in. That’s task completion under autonomy — the same thing a real eval harness has to score for any production agent, security or not. The cost line is the honest part of the writeup: $1,500 of tokens is absurdly cheap next to a human pentester’s day rate, and the agent doesn’t get bored on attempt forty.

The asymmetry is what keeps me up. A defender has to harden every path; an autonomous agent only has to find one, and it will grind through token budget tirelessly until it does. We’ve spent a decade telling ourselves that “attacker has to be right once” was a human-throughput problem — bounded by how many analysts you could hire. Remove that bound and the economics invert.

So the interesting follow-up isn’t “can LLMs hack” — the HN thread already litigates that. It’s whether our eval harnesses are built to measure multi-step autonomous exploitation at all, or whether we’re still grading models on flashcards. Which side does cheap, tireless agentic search actually favor first — the red team or the blue team?