让大模型学会识别并解决不确定问题，而非简单说‘不知道’。
Do not Abstain! Identify and Solve the Uncertainty
- 构建新基准ConfuseBench，区分三类不确定性：文档不足、能力有限、问题模糊。
- 发现现有模型常误判原因，将问题归因于模糊提问，忽视自身能力缺陷。
- 提出交互式优化方法InteractDPO，提升模型生成精准追问的能力。
尽管大型语言模型在多个领域广泛应用，但在面对不确定性时往往表现出过度自信。现有解决方案多依赖回避性回应（如“我不知道”），忽略了识别与应对不确定性的机会。为系统研究并提升模型识别和解决不确定性来源的能力，我们提出ConfuseBench基准，聚焦三类不确定性：文档稀缺性、能力局限性和查询模糊性。实验表明，当前大模型难以准确识别不确定性的根本原因，更倾向于将责任归于查询模糊，尤其在较弱模型中表现明显。为此，我们首先生成上下文感知的追问，突出原始问题的困惑点；然后基于追问回答的独特性判断不确定性来源；最后采用基于策略的训练方法InteractDPO生成更优追问。实验验证了该方法的有效性。
原文摘要 · Abstract (English)
Despite the widespread application of Large Language Models (LLMs) across various domains, they frequently exhibit overconfidence when encountering uncertain scenarios, yet existing solutions primarily rely on evasive responses (e.g., "I don't know") overlooks the opportunity of identifying and addressing the uncertainty to generate more satisfactory responses. To systematically investigate and improve LLMs' ability of recognizing and addressing the source of uncertainty, we introduce \textbf{ConfuseBench}, a benchmark mainly focus on three types of uncertainty: document scarcity, limited capability, and query ambiguity. Experiments with ConfuseBench reveal that current LLMs struggle to accurately identify the root cause of uncertainty and solve it. They prefer to attribute uncertainty to query ambiguity while overlooking capability limitations, especially for those weaker models. To tackle this challenge, we first generate context-aware inquiries that highlight the confusing aspect of the original query. Then we judge the source of uncertainty based on the uniqueness of the inquiry's answer. Further we use an on-policy training method, InteractDPO to generate better inquiries. Experimental results demonstrate the efficacy of our approach.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。