让大模型学会识别自己错在哪，用自信度精准检测错误。
Know When You're Wrong: Aligning Confidence with Correctness for LLM Error Detection
- 用输出锚点概率构建归一化置信度，无需外部验证。
- 微调后置信度与正确率匹配度提升，校准误差降至0.034。
- 适合需要高可信度的决策系统，如医疗、金融场景。
随着大型语言模型（LLMs）在关键决策系统中的广泛应用，缺乏可靠的不确定性度量带来了根本性的可信风险。本文提出一种基于输出锚点词概率的归一化置信度评分：结构化任务使用分类标签，开放生成任务使用自评估回答（是/否）。该方法可直接检测错误与幻觉，开销极小且无需外部验证。主要贡献有三：其一，提出归一化置信度与自评估框架，在七个不同基准任务和五种不同架构与规模的LLM上实现可靠的置信度估计；其二，理论分析表明，监督微调（SFT）通过最大似然估计获得良好校准的置信度，而强化学习方法（PPO、GRPO）及DPO因奖励利用导致过度自信；其三，提出后强化学习的SFT自蒸馏方法，恢复强化学习训练模型的置信度可靠性。实证结果显示，对Qwen3-4B模型，SFT使平均置信度-正确率AUROC从0.806升至0.879，校准误差从0.163降至0.034，而GRPO与DPO则削弱了置信度可靠性。通过自适应检索增强生成（RAG）验证其实际价值：仅使用58%的检索操作，即可在TriviaQA上恢复95%的最大准确率增益。
原文摘要 · Abstract (English)
As large language models (LLMs) are increasingly deployed in critical decision-making systems, the lack of reliable methods to measure their uncertainty presents a fundamental trustworthiness risk. We introduce a normalized confidence score based on output anchor token probabilities: classification labels for structured tasks and self-evaluation responses (Yes/No) for open-ended generation. This enables direct detection of errors and hallucinations with minimal overhead and without external validation. We make three key contributions. First, we propose a normalized confidence score and self-evaluation framework that exposes reliable confidence estimates for error detection across seven diverse benchmark tasks and five LLMs of varying architectures and sizes. Second, our theoretical analysis reveals that supervised fine-tuning (SFT) yields well-calibrated confidence through maximum-likelihood estimation, whereas reinforcement learning methods (PPO, GRPO) and DPO induce overconfidence via reward exploitation. Third, we propose post-RL SFT with self-distillation to restore confidence reliability in RL-trained models. Empirical results demonstrated that SFT improved average confidence-correctness AUROC from 0.806 to 0.879 and reduced calibration error from 0.163 to 0.034 on Qwen3-4B, while GRPO and DPO degraded confidence reliability. We demonstrated practical value through adaptive retrieval-augmented generation (RAG) that selectively retrieves context when the model lacks confidence, using only 58\% of retrieval operations to recover 95\% of the maximum achievable accuracy gain on TriviaQA
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。