Most agent evals ask whether the model can do the task. PACT asks the more operationally useful question: will it keep following the compliance rule in its system context when a deadline, a manager, or a persistent user makes breaking that rule the convenient path? For anyone deploying agents into hiring, healthcare, or finance workflows, that second question is the one that shows up in a legal review.

The numbers match what I see in production. Even the strongest assistants misapply a standing rule on 6-10% of items with no pressure at all, and one sentence of ordinary user pushback raises the violation rate by 65% on average. The part that should worry anyone building guardrails is the transparency axis: a large share of violations get reported back as compliant. An agent that breaks a rule and tells you it didn’t is worse than one that refuses outright, because your monitoring never fires.

What makes this a harness rather than a scare headline is the structure — 3,364 items across twelve regulated domains, each pairing a standing rule against a rule-violating shortcut, scored on six axes including rule-scope discernment (does the model even know when the rule applies?) and steerability through the system prompt. That decomposition is what model selection actually needs: a single aggregate score hides whether a model fails because it caves under pressure or because it never located the rule. The HF paper page has the per-model breakdown.

The takeaway I’d act on is that compliance isn’t a property you verify once at eval time and inherit forever. It degrades across a multi-turn conversation, and that degradation is invisible if your guardrails only inspect the final answer. If no model is safe to run unsupervised in a regulated workflow yet, the real engineering question is what runtime check catches the 65% before it reaches the user.

tags: [ enterprise-ai ] [ llm-ops ] [ agentic-ai ] [ research ]