让AI模型学会在不确定时放弃回答，提升问答可靠性。
To Answer or to Abstain: Mitigating Search-Agent Hallucinations via Abstention-Aware Reinforcement Learning
- 引入可动态调整的拒答奖励机制，结合模型自身能力评估
- 在失败检索时仍保持10.3%精度提升，RA-F1指标提高2.9%
- 适合对可靠性要求高的开放域问答场景
近期将大语言模型（LLMs）与搜索工具结合，并采用基于结果反馈的强化学习（RL）方法，在开放域问答任务上取得了新进展。然而，我们指出当前训练范式存在关键缺陷：仅奖励正确答案，未惩罚检索失败时的虚构回答，反而加剧了幻觉问题。为此，我们提出拒答感知强化学习（AWA-RL），通过模型查询特定先验能力与持续的在线策略观测，动态调节拒答奖励。我们还引入新指标RA-F1，衡量能力与可靠性之间的权衡。相比不支持拒答的基线，AWA-RL在精度上绝对提升最高达10.3%，整体RA-F1提升2.9%，且原始准确率仅轻微下降。结果表明，AWA-RL成功训练出兼具高能力与高可靠性的搜索代理。代码、数据及模型权重已公开于https://github.com/zfj1998/AWA-RL。
原文摘要 · Abstract (English)
Recent advances in equipping Large Language Models (LLMs) with search tools and outcome-reward reinforcement learning (RL) have achieved new state-of-the-art results on open-domain QA tasks. However, we argue that current training paradigms harbor a critical vulnerability: they predominantly reward correct answers but fail to penalize fabricated ones when retrieval fails, thereby implicitly exacerbating hallucinations. To address this, we propose Abstention-Aware Reinforcement Learning (AWA-RL), which dynamically shapes the abstention reward utilizing the model's query-specific prior capabilities and continuous on-policy training observations. We also introduce a novel metric, RA-F1, to measure the capability-reliability trade-off. Compared to non-abstaining baselines, AWA-RL boosts absolute precision by up to 10.3% and overall RA-F1 by 2.9%, with only marginal sacrifice in raw accuracy. These results confirm that AWA-RL successfully yields highly capable and reliable search agents. The code, data, and model weights are publicly available at https://github.com/zfj1998/AWA-RL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。