arXiv:2608.12008cs.CL2026-08

让大模型只答有把握的问题,还能保证答错率可控。

Asymptotic Risk Calibration for Selective Question Answering

  • 用统计方法校准不确定性,动态调整接受答案的门槛。
  • 在两个数据集上既减少错误答案,又保留更多有效回答。
  • 不需重新训练,可搭配多种不确定度估计方式,通用性强。

大语言模型可能生成流畅但错误的答案,因此量化不确定性对可靠问答至关重要。现有启发式不确定性评分无法完美区分正确与错误预测,固定阈值也无法控制被接受答案的错误率。为此,我们提出A-CRC-QA,一种后处理校准框架,将选择条件下的误差控制转化为线性期望约束,并采用受单调化启发的基于经验风险的校准方法。由于实例级损失通常不随接受阈值单调变化,该框架关注渐近风险控制而非有限样本控制。A-CRC-QA具有模型无关性,无需额外训练,可与不同不确定性估计器结合。在CoQA和MedMCQA上的实验表明,其在开放与闭合问答中均表现良好,相比未校准及基于置信度边界的基线,在接受答案的可靠性与保留率之间取得了更优权衡。

原文摘要 · Abstract (English)

Large language models (LLMs) may generate fluent but incorrect answers, making uncertainty quantification important for reliable question answering. However, heuristic uncertainty scores cannot perfectly distinguish correct predictions from incorrect ones, and directly applying a fixed uncertainty threshold provides no statistical control over the error rate among accepted answers. To address this limitation, we propose A-CRC-QA, a post-hoc calibration framework for uncertainty-aware selective question answering. The proposed method reformulates selection-conditioned error control as a linear expectation constraint and applies a monotonized empirical-risk calibration procedure inspired by conformal risk control. Since the resulting instance-wise loss is generally non-monotone with respect to the acceptance threshold, our framework targets asymptotic rather than finite-sample risk control. A-CRC-QA is model-agnostic, requires no additional training, and can be combined with different uncertainty estimators. Experiments on CoQA and MedMCQA demonstrate its applicability to both open-ended and closed-ended question answering, achieving a favorable trade-off between accepted-answer reliability and answer retention compared with uncalibrated and confidence-bound-based baselines.

不确定性量化问答系统风险校准大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。