用二值检索增强奖励，让大模型生成更真实内容而不丢能力。
Train for Truth, Keep the Skills: Binary Retrieval-Augmented Reward Mitigates Hallucinations
- 采用二值化奖励机制，仅当输出完全正确时才给奖励
- 开放生成任务中幻觉率降低39.3%，问答任务错误减少超20%
- 保持指令遵循、数学与代码生成能力，适合高可靠性场景
语言模型常生成训练数据未支持的虚假信息，称为外在幻觉。现有缓解方法常损害开放生成与下游任务表现，限制实用性。本文提出一种在线强化学习方法，使用新型二值检索增强奖励（RAR）。不同于连续奖励，本方法仅在模型输出完全事实正确时给予1分奖励，否则为0。我们在Qwen3推理模型上评估该方法，涵盖多种任务。在开放生成任务中，二值RAR使幻觉率降低39.3%，显著优于监督训练与连续奖励强化学习基线。在短文本问答中，模型学会合理拒答，面对知识不足时主动输出“我不知道”，使PopQA与GPQA错误率分别减少44.4%和21.7%。关键在于，这些事实性提升未导致指令遵循、数学或代码生成性能下降；而连续奖励强化学习虽提升事实性，却引发质量退化。
原文摘要 · Abstract (English)
Language models often generate factually incorrect information unsupported by their training data, a phenomenon known as extrinsic hallucination. Existing mitigation approaches often degrade performance on open-ended generation and downstream tasks, limiting their practical utility. We propose an online reinforcement learning method using a novel binary retrieval-augmented reward (RAR) to address this tradeoff. Unlike continuous reward schemes, our approach assigns a reward of one only when the model's output is entirely factually correct, and zero otherwise. We evaluate our method on Qwen3 reasoning models across diverse tasks. For open-ended generation, binary RAR achieves a 39.3% reduction in hallucination rates, substantially outperforming both supervised training and continuous-reward RL baselines. In short-form question answering, the model learns calibrated abstention, strategically outputting "I don't know" when faced with insufficient parametric knowledge. This yields 44.4% and 21.7% fewer incorrect answers on PopQA and GPQA, respectively. Crucially, these factuality gains come without performance degradation on instruction following, math, or code, whereas continuous-reward RL, despite improving factuality, induces quality regressions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。