让大模型主动表达不确定,减少错误自信。
LLMs Should Express Uncertainty Explicitly

- 训练模型在推理中或结尾显式标记不确定
- 显著降低错误自信率,提升答案准确率
- 适合需要可靠决策的AI应用如医疗、金融
大型语言模型常给出自信但错误的答案,可能导致实际应用中的高风险失误。本文研究后训练是否能让模型在自身回答中明确表达自我评估:当模型不确定时,能否被训练为在回答中发出信号?核心设计问题在于信号应出现在推理过程中还是答案生成后。针对答案末尾的自我评估,我们训练模型对回答输出置信度分数,目标是正确答案高置信、错误答案低置信;针对推理过程中的自我评估,我们训练模型在当前推理状态不可靠时输出标记<uncertain>。在事实推理任务中,两种方式均显著减少过度自信错误,同时提升答案质量,并可作为检索增强生成(RAG)的触发条件。进一步分析显示,末尾置信度表达强化了预训练模型中已存在的置信结构,而推理中<uncertain>标记则教会模型识别高风险推理步骤,参数变化集中在模型深层。
原文摘要 · Abstract (English)
Large language models (LLMs) often produce confident yet incorrect answers, which can lead to risky failures in real-world applications. We study whether post-training can make a model's self-assessment explicit: when the model is uncertain, can it be trained to signal so within its own response? A central design question is where in the response this signal should be exposed -- during reasoning, while the answer is still being formed, or at the end, once the answer has been produced. We study both. For end-of-reasoning self-assessment, we train the model to verbalize a confidence score for its response, with the aim of high confidence on correct answers and low confidence on incorrect ones. For during-reasoning self-assessment, we train the model to emit the marker <uncertain> whenever its current reasoning state appears unreliable. Across factual reasoning tasks, both forms sharply reduce overconfident errors while improving answer quality, and both can be used as triggers for retrieval augmented generation (RAG) to improve the final response. We further analyze their internal mechanisms: end-of-reasoning verbalized confidence sharpens a confidence-related structure already present in the pretrained model, whereas during-reasoning <uncertain> emission teaches the model to mark high-risk reasoning steps, with parameter changes concentrated in the model's late layers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。