让大模型学会说‘不知道’,用惩罚机制提升可信度
Honesty over Accuracy: Trustworthy Language Models through Reinforced Hesitation
- 用三元奖励机制惩罚错误回答,鼓励模型在不确定时选择不答
- 在逻辑题测试中,不同惩罚强度生成从激进到保守的多版本模型
- 通过自我重试和级联推理,实现低成本高可信的决策协作
现代语言模型缺乏可信智能的核心能力:知道何时不应回答。尽管在基准测试中表现准确,这些模型仍会自信地产生幻觉,即使错误答案可能导致灾难性后果。我们在GSM8K、MedQA和GPQA上的评估显示,前沿模型几乎从不主动放弃回答,即使有严重惩罚警告,表明提示无法克服训练中‘任何回答优于无回答’的激励。为此,我们提出强化犹豫(Reinforced Hesitation, RH):对可验证奖励的强化学习(RLVR)进行改进,采用三元奖励(+1正确,0放弃,-λ错误)。在逻辑谜题上的受控实验表明,调整λ值可生成沿帕累托前沿分布的多种模型,每种对应特定风险环境下的最优策略:低惩罚生成激进回答者,高惩罚生成保守放弃者。我们进一步引入两种推理策略,利用训练出的放弃行为作为协调信号:级联将查询依次传递给风险容忍度递减的模型;自级联则在放弃时重新查询同一模型。两者均以更低计算成本超越多数投票。结果证明,放弃应作为首要训练目标,使‘我不知道’从失败转为可信的协调信号,从而实现模型基于认知边界的诚实表达。
原文摘要 · Abstract (English)
Modern language models fail a fundamental requirement of trustworthy intelligence: knowing when not to answer. Despite achieving impressive accuracy on benchmarks, these models produce confident hallucinations, even when wrong answers carry catastrophic consequences. Our evaluations on GSM8K, MedQA and GPQA show frontier models almost never abstain despite explicit warnings of severe penalties, suggesting that prompts cannot override training that rewards any answer over no answer. As a remedy, we propose Reinforced Hesitation (RH): a modification to Reinforcement Learning from Verifiable Rewards (RLVR) to use ternary rewards (+1 correct, 0 abstention, -$λ$ error) instead of binary. Controlled experiments on logic puzzles reveal that varying $λ$ produces distinct models along a Pareto frontier, where each training penalty yields the optimal model for its corresponding risk regime: low penalties produce aggressive answerers, high penalties conservative abstainers. We then introduce two inference strategies that exploit trained abstention as a coordination signal: cascading routes queries through models with decreasing risk tolerance, while self-cascading re-queries the same model on abstention. Both outperform majority voting with lower computational cost. These results establish abstention as a first-class training objective that transforms ``I don't know'' from failure into a coordination signal, enabling models to earn trust through calibrated honesty about their limits.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。