让大模型在干扰下保持稳定，从而更真实地反映自信程度。
CaliDist: Calibrating Large Language Models via Behavioral Robustness to Distraction
- 通过引入语义干扰测试模型稳定性，动态调整初始置信度。
- 平均将预期校准误差从23%降至7%，相对提升70%。
- 适合关注模型可信度、对抗干扰能力的研究者使用。
现有大语言模型校准方法常忽视一个关键信任维度：模型对无关或误导信息的抗干扰行为鲁棒性。本文认为，模型的真实置信度应反映其在认知压力下的稳定性。我们提出CaliDist，一种新型后处理校准方法，直接测量并惩罚模型对干扰的敏感性。CaliDist量化当输入提示被加入语义干扰时，大模型预测与不确定性变化的程度，以此信号自适应地调整初始置信度。在六个不同大模型和七个自然语言理解分类基准上的实验表明，CaliDist始终优于强基线，平均将预期校准误差（ECE）从23%降至7%，相对改善70%，证明行为稳定性是强大的校准信号。代码与数据集已开源于github.com/anas-jawad/CaliDist。
原文摘要 · Abstract (English)
Existing calibration methods for Large Language Models (LLMs) often overlook a critical dimension of trustworthiness: a model's behavioral robustness to irrelevant or misleading information. In this paper, we argue that a model's true confidence should reflect its stability under cognitive pressure. We introduce CaliDist, a novel post-hoc calibration approach that directly measures and penalizes a model's susceptibility to distraction. CaliDist quantifies how an LLM's predictions and uncertainty change when its input prompt is perturbed with semantic distractors. This stability (or lack thereof) signal is then used to adaptively scale the model's initial confidence score. Our extensive experiments on seven Natural Language Understanding classification benchmarks using six distinct LLMs show that CaliDist consistently achieves lower Expected Calibration Error (ECE) and Brier Score compared with strong baselines. Remarkably, our method reduces the ECE from 23% to 7% on average--a relative improvement of 70%--demonstrating that behavioral stability is a powerful signal for calibration. We make our code and datasets available at github.com/anas-jawad/CaliDist.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。