If you run agents against a third-party inference endpoint, you’re trusting that the model behind the URL is the one you benchmarked — and that it stays that model at 2am under load. Ventor-QTest is a black-box audit that stops you from taking that on faith, and it needs no probability information from the provider.
It treats hosted model routing as a stochastic process and probes it two ways:
- 🔍 Repeated-request fidelity — fire the same frozen context many times, rebuild the output distribution from raw text counts, and report average fidelity loss (AFL) as a null-bias-corrected, KL-style drift statistic
- ⚡ Long-sequence fidelity — independent runs surface extreme fidelity loss (EFL) in the upper tail, catching rare-but-severe deviations a mean would smooth over
- 📊 The text-only estimate holds up — AFL tracks a logprob-derived comparator closely on routes where you can actually check it
- ⚠️ EFL, not accuracy, predicts the damage — it barely moves GPQA-Diamond scores, yet high EFL coincides with falling Terminal-Bench pass rates as task exposure grows
That last point is the one I’d take straight to production. A provider silently swapping in a quantized or rerouted model can look fine on short QA while quietly wrecking long-horizon agentic runs — exactly the setting where you’re least able to eyeball each output. The HF paper page links the open-source implementation if you want to point it at your own vendors.
We pour effort into eval harnesses for our own models and spend almost nothing verifying that the API we rent still serves what the contract says. Which of your production routes could pass a short-prompt eval today and still be drifting under long agentic load?