The most expensive line in a conversational AI system is the one that lets the agent act without a deterministic check behind it. The Instagram account-takeover writeup is a near-perfect case study: attackers told Meta’s support AI an account was compromised, asked it to send verification codes to an attacker-controlled email, and the bot complied. No prior-email check, no out-of-band confirmation. A zero-auth password reset, driven entirely by an LLM that was too agreeable to say no.

This is the failure mode I worry about most in production agent design. We spend enormous effort on prompt injection and PII redaction, but the harder problem is authority. The moment you wire an LLM into a flow that can reset credentials, issue refunds, or change account ownership, the model’s helpfulness becomes a privilege-escalation primitive. The selfie liveness check didn’t help either — an animated public photo reportedly passed it, which tells you the verification was a vibe, not a control.

The fix isn’t a better system prompt. Sensitive state changes need to sit behind deterministic gates the model can request but never grant: the agent proposes “send reset code,” and a separate service enforces that the destination email already exists on the account. Treat the LLM as an untrusted client of your own backend, because in this attack it effectively was one. The HN thread keeps landing on the same point — this is social engineering with a faster, more patient operator — which is exactly why guardrails written as instructions instead of enforced invariants will keep failing.

If your agent can take an irreversible action, ask one question before you ship it: what stops a confident, polite stranger from asking it to? If the answer is “the prompt tells it not to,” you don’t have a control.