RiskChainBench is framed as an abuse-detection benchmark, but the useful result is where it says the pipeline breaks. Platform abuse hides redirection in emojis, homophones, and character decomposition, then routes people to fraud or gambling destinations. The benchmark splits the job in two: first restore the obfuscated message, intent, and destination, then have the same model act as a VLM-driven web agent that investigates the real site and files an evidence-cited risk report — with no domain-reputation shortcuts allowed.
Decoding the message turns out not to be the hard part. Across ten models, entry-recovery Top-1 ranges from 35% to 95%, but web-decision accuracy tops out at 63% and bottoms at 26%. The tell is in the failure breakdown: execution failures account for 31.9% of web runs, while post-decision typing errors are 0.9%. Models that can read through the obfuscation still fall apart when they have to explore a live site and hold a risk judgment together — the agentic half, not the language half.
That maps cleanly onto how I think about eval harnesses for anything guardrail-adjacent. A restoration score tells you the model understood the input; it says nothing about whether the downstream agent can act on that understanding without derailing. Composing the two stages offline — gating the web investigation on the frozen entry prediction — is the right instinct, because it stops a strong decoder from masking a weak investigator inside one blended number.
The HF paper page has the full protocol and the resettable local sandbox they released. If your safety eval only measures the classification step, what is your agent’s real-world execution failure rate — and would you know?