让AI学会在不确定时选择问、答或放弃,避免胡说八道。
PassiveQA: A Three-Action Framework for Epistemically Calibrated Question Answering via Supervised Finetuning
- 设计三动作框架:答、问、弃,引导模型判断信息是否足够。
- 在多个数据集上提升拒答召回率,降低幻觉率,宏平均F1显著提高。
- 适合需要高可靠性问答的场景,如医疗、法律等关键领域。
大型语言模型在问答和检索增强生成(RAG)中表现优异,但默认用户问题完整且可答。现实中问题常不完整、模糊或缺少关键变量,导致模型产生过度自信或虚构的回答。本文研究不完备信息下的决策感知查询解决,要求模型判断是否回答、追问澄清或放弃。实验表明,标准及增强型RAG系统缺乏这种认知意识,常在信息不足时仍强行作答。为此,我们提出PassiveQA,一种通过监督微调实现的三动作框架。该方法融合结构化信息状态表示、知识图谱支撑的上下文,以及一个微调后的规划器,显式建模缺失变量与决策推理过程。在多个问答数据集上的实验显示,微调后的规划器在计算受限训练条件下显著提升了宏平均F1和拒答召回率,同时降低了幻觉率。结果有力证明,认知决策能力必须在训练阶段学习,而非推断时硬加。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have achieved strong performance in question answering and retrieval-augmented generation (RAG), yet they implicitly assume that user queries are fully specified and answerable. In real-world settings, queries are often incomplete, ambiguous, or missing critical variables, leading models to produce overconfident or hallucinated responses. In this work, we study decision-aware query resolution under incomplete information, where a model must determine whether to Answer, Ask for clarification, or Abstain. We show that standard and enhanced RAG systems do not reliably exhibit such epistemic awareness, defaulting to answer generation even when information is insufficient. To address this, we propose PassiveQA, a three-action framework that aligns model behaviour with information sufficiency through supervised finetuning. Our approach integrates structured information-state representations, knowledge graph-grounded context, and a finetuned planner that explicitly models missing variables and decision reasoning. Experiments across multiple QA datasets show that the finetuned planner achieves significant improvements in macro F1 and abstention recall while reducing hallucination rates, under a compute-constrained training regime. These results provide strong empirical evidence that epistemic decision-making must be learned during training rather than imposed at inference time.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。