arXiv:2605.12446cs.LGcs.CL2026-05

让大模型的自信表达更真实,通过分离回答与自信评估提升可靠性。

ORCE: Order-Aware Alignment of Verbalized Confidence in Large Language Models

论文配图:ORCE: Order-Aware Alignment of Verbalized Confidence in Large Language Models
图 1 · 摘自论文原文
  • 先生成答案,再基于固定问答对估算自信,避免干扰答案质量。
  • 用多轮生成构建可信度排序,优化自信表述的相对顺序。
  • 在推理和知识密集型任务上显著提升错误预测能力,适合部署场景。

大语言模型常在错误时仍表现出高确定性,可靠的信心估计对实际应用至关重要。口头化信心(verbalized confidence)让模型以自然语言表达信心,提供灵活且用户友好的不确定性信号,即使在无法获取词元概率时也可使用。然而,现有方法通常联合优化答案生成与信心生成,导致信心校准目标干扰答案准确性。本文提出一种解耦且有序感知的口头化信心校准框架:先生成答案,再基于固定的问答对估计信心,使信心优化不直接扰动答案生成过程。为对齐信心与正确性概率,我们构建基于采样的替代指标,利用基于排名的强化学习目标,促使更高可信度响应获得更高口头化信心。在推理与知识密集型基准测试中,该方法显著提升校准性能与错误预测能力,同时基本保持答案准确性。结果表明,通过解耦信心估计与答案生成,并优化响应间信心的相对顺序,可更可靠地对齐口头化信心。

原文摘要 · Abstract (English)

Large language models (LLMs) often produce answers with high certainty even when they are incorrect, making reliable confidence estimation essential for deployment in real-world scenarios. Verbalized confidence, where models explicitly state their confidence in natural language, provides a flexible and user-facing uncertainty signal that can be applied even when token logits are unavailable. However, existing verbalized-confidence methods often optimize answer generation and confidence generation jointly, which can cause confidence-alignment objectives to interfere with answer accuracy. In this work, we propose a decoupled and order-aware framework for verbalized confidence calibration. Our method first generates an answer and then estimates confidence conditioned on the fixed question--answer pair, allowing confidence optimization without directly perturbing the answer-generation process. To align confidence with correctness likelihood, we construct a sampling-based surrogate from multiple model completions and optimize rank-based reinforcement learning objectives that encourage responses with higher estimated correctness likelihood to receive higher verbalized confidence. Experiments on reasoning and knowledge-intensive benchmarks show that our method improves calibration and failure prediction performance while largely preserving answer accuracy. These results demonstrate that verbalized confidence can be more reliably aligned by decoupling confidence estimation from answer generation and optimizing the relative ordering of confidence across responses.

大模型信心估计解耦设计强化学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。