强化微调让大模型更自信地胡说，但加点特殊数据就能让它学会拒绝不懂的问题。
The Hallucination Tax of Reinforcement Finetuning
- 用合成数学题测试模型拒答能力，发现强化微调使拒答率下降超80%
- 在微调中加入10%特殊数据，拒答行为基本恢复，解题准确率几乎不变
- 让模型学会识别自身知识边界，提升对陌生问题的泛化能力
强化微调（RFT）已成为提升大语言模型（LLMs）推理能力的标准方法，但其对模型可信度的影响尚未充分研究。本文揭示并系统分析了RFT的一个关键副作用——‘幻觉税’：模型拒绝回答不可答问题的能力显著下降，导致其在面对无法解答的问题时仍自信地产生幻觉答案。为此，我们提出了SUM（Synthetic Unanswerable Math）数据集，该数据集包含高质量的不可答数学问题，用于探测模型基于信息不足或模糊内容判断问题不可答的能力。实验表明，标准RFT训练可使模型拒答率下降超过80%，显著增加幻觉倾向。进一步研究表明，在RFT中仅引入10% SUM数据即可显著恢复合理的拒答行为，且对可解任务的准确率影响极小。关键在于，该方法使模型能够利用推理时的计算资源，自主评估自身不确定性与知识边界，不仅提升对域外数学问题的泛化能力，也改善事实类问答的表现。
原文摘要 · Abstract (English)
Reinforcement finetuning (RFT) has become a standard approach for enhancing the reasoning capabilities of large language models (LLMs). However, its impact on model trustworthiness remains underexplored. In this work, we identify and systematically study a critical side effect of RFT, which we term the hallucination tax: a degradation in refusal behavior causing models to produce hallucinated answers to unanswerable questions confidently. To investigate this, we introduce SUM (Synthetic Unanswerable Math), a high-quality dataset of unanswerable math problems designed to probe models' ability to recognize an unanswerable question by reasoning from the insufficient or ambiguous information. Our results show that standard RFT training could reduce model refusal rates by more than 80%, which significantly increases model's tendency to hallucinate. We further demonstrate that incorporating just 10% SUM during RFT substantially restores appropriate refusal behavior, with minimal accuracy trade-offs on solvable tasks. Crucially, this approach enables LLMs to leverage inference-time compute to reason about their own uncertainty and knowledge boundaries, improving generalization not only to out-of-domain math problems but also to factual question answering tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。