Every production agent already makes a confidence decision on every turn — it just usually makes it badly. Retry? Escalate to a human? Ship the answer? Most of us wire that gate to token probabilities or run self-consistency and count how often the samples agree. XConf argues the current inference is the wrong place to look, and the argument lands: a model resampling its own reasoning tends to be confident about exactly the things it is confidently wrong about.
The move is to estimate confidence against a memory of graded past episodes — each one storing the task, the model’s reflection, its stated confidence, the outcome, and a lesson written after the grade arrived. On a new task it recalls similar past episodes met with a similar stated confidence, reads off their historical success rate, and restates its confidence informed by its own record. That is the same instinct behind good LLM ops: don’t trust a self-report, check it against what actually happened last time.
What makes it worth a production look is the operating cost. It needs no logit access and no weight updates, works across output formats, and costs one generation — a tenth of ten-sample self-consistency — while matching or beating that baseline on AUROC in 23 of 24 comparisons, with lower calibration error. The number I would actually chase: abstaining on the 10% least-confident agent episodes lifts delivered success by up to 8.7 points. That is a selective-prediction knob you can turn without retraining anything.
The catch is the memory itself. An experiential estimator is only as calibrated as its episode log, and a log that drifts as your tools and prompts change becomes its own observability problem. The HF paper page has the benchmark spread. Would you trust an agent’s confidence more if it were graded on its own history than on its own logits?