区分模型不确定性和数据不确定性，让大模型更聪明地拒绝回答
Beyond "I Don't Know": Evaluating LLM Self-Awareness in Discriminating Data and Model Uncertainty
- 设计3500+题基准测试，区分输入模糊和模型能力不足
- 顶尖模型虽答对率高，但分不清拒绝原因
- 轻量方法提升识别能力，适合需要精准拒答的应用
可靠的大型语言模型应在置信度不足时选择不回答。然而，以往研究常将拒绝统一视为‘我不知道’，未能区分输入本身的模糊性（数据不确定性）与模型自身能力限制（模型不确定性）。这种混淆影响下游决策，如是否请求澄清或调用外部工具。本文提出UA-Bench，一个包含超过3500个问题的基准，覆盖六大数据集，涵盖知识密集型与推理密集型任务，用于评估模型对不确定性的显式归因能力。对18个前沿LLM的评估显示，即使最先进的模型也难以可靠区分两类不确定性，且高回答准确率并不意味着强不确定性归因能力。为此，我们提出一种轻量级数据合成与强化学习策略。在Qwen3-4B-Instruct-2507和Qwen3-8B的思维模式下实验表明，该方法在保持回答准确性的同时显著提升了不确定性归因能力。代码与数据已公开。
原文摘要 · Abstract (English)
Reliable Large Language Models (LLMs) should abstain when confidence is insufficient. However, prior studies often treat refusal as a generic "I don't know'', failing to distinguish input-level ambiguity (data uncertainty) from capability limitations (model uncertainty). This lack of distinction limits downstream action decisions like requesting clarification or invoking external tools. In this work, we introduce UA-Bench, a benchmark of over 3,500 questions drawn from six datasets spanning knowledge-intensive and reasoning-intensive tasks, designed to evaluate explicit uncertainty attribution. An evaluation of 18 frontier LLMs shows that even state-of-the-art models struggle to reliably discriminate between data uncertainty and model uncertainty, and that high answer accuracy does not necessarily imply strong uncertainty attribution ability. To narrow this gap, we propose a lightweight data synthesis and reinforcement learning strategy. Experiments on both Qwen3-4B-Instruct-2507 and Qwen3-8B in thinking mode show that the proposed method improves uncertainty attribution while preserving answer accuracy. Our code and data are publicly available now.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。