让大模型学会说'不知道',减少幻觉错误。
Rewarding Intellectual Humility Learning When Not To Answer In Large Language Models
- 用可验证奖励机制训练模型,鼓励在不确定时选择不回答。
- 适度的拒答奖励(约-0.25至0.3)能显著降低错误率且不影响准确率。
- 适合关注模型可靠性与事实性的人士,尤其在医疗、数学等高风险领域。
大语言模型常生成幻觉或无法验证的内容,影响其在事实类任务中的可信度。本文研究基于可验证奖励的强化学习(RLVR)训练范式,通过明确奖励拒答('我不知道')来促进模型的智力谦逊。我们在MedMCQA和Hendrycks Math基准上对Granite-3.3-2B-Instruct和Qwen-3-4B-Instruct进行微调与评估,采用三元奖励结构(-1, r_abs, 1),在不同拒答奖励设置下测试表现。进一步研究了在强化学习前先通过监督微调教授拒答策略的效果。结果表明,适度的拒答奖励(r_abs ≈ -0.25 至 0.3)在多项选择任务中持续降低错误响应,且未造成严重准确率下降,更大模型对拒答激励更具鲁棒性。在开放问答任务中,受限于探索不足,效果有限,但可通过监督拒答训练部分缓解。总体表明,可验证奖励设计是应对模型幻觉的可行且灵活方案。代码已开源:https://github.com/Mystic-Slice/rl-abstention。
原文摘要 · Abstract (English)
Large Language Models (LLMs) often produce hallucinated or unverifiable content, undermining their reliability in factual domains. This work investigates Reinforcement Learning with Verifiable Rewards (RLVR) as a training paradigm that explicitly rewards abstention ("I don't know") alongside correctness to promote intellectual humility. We fine-tune and evaluate Granite-3.3-2B-Instruct and Qwen-3-4B-Instruct on the MedMCQA and Hendrycks Math benchmarks using a ternary reward structure ($-1$, r_abs, 1) under varying abstention reward structures. We further study the effect of combining RLVR with supervised fine-tuning strategies that teach abstention prior to reinforcement learning. Our results show that moderate abstention rewards (r_abs $\approx -0.25$ to 0.3) consistently reduce incorrect responses without severe accuracy degradation on multiple-choice tasks, with larger models exhibiting greater robustness to abstention incentives. On open-ended question answering, we observe limitations due to insufficient exploration, which can be partially mitigated through supervised abstention training. Overall, these findings demonstrate the feasibility and flexibility of verifiable reward design as a practical approach for hallucination mitigation in language models. Reproducible code for our abstention training framework is available here https://github.com/Mystic-Slice/rl-abstention.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。