arXiv:2512.19920cs.LGcs.AI2025-12被引 11

让大模型学会说‘不知道’,有效减少胡说八道。

Mitigating LLM Hallucination via Behaviorally Calibrated Reinforcement Learning

  • 用奖励机制鼓励模型在不确定时主动放弃回答
  • 小模型在不确定识别上超越顶尖大模型
  • 适合需要高可信度的医疗、金融等关键场景

大模型在关键领域部署受限于持续存在的幻觉问题——生成看似合理但事实错误的内容。尽管规模定律推动了通用能力提升,理论表明幻觉并非随机误差,而是训练目标偏向模仿数据分布而非追求认知诚实的可预测结果。传统强化学习验证(RLVR)使用二元奖励信号,无意中激励模型像优秀应试者一样猜测,只要正确概率大于零就强行作答。本文系统研究行为校准机制,通过优化严格适当的评分规则,使模型在无把握时随机选择不回答或标记具体可疑陈述,从而实现行为与准确率对齐。基于 Qwen3-4B-Instruct 的实证分析显示,经过行为校准的强化学习使小型模型在不确定性量化方面超越前沿模型。在数学推理任务(BeyondAIME)中,其对数尺度上的准确率-幻觉比增益达0.806,超过 GPT-5 的0.207;在跨领域事实问答(SimpleQA)中,4B 模型零样本校准误差与 Grok-4、Gemini-2.5-Pro 等领先模型相当,尽管其事实准确率较低。

原文摘要 · Abstract (English)

LLM deployment in critical domains is currently impeded by persistent hallucinations--generating plausible but factually incorrect assertions. While scaling laws drove significant improvements in general capabilities, theoretical frameworks suggest hallucination is not merely stochastic error but a predictable statistical consequence of training objectives prioritizing mimicking data distribution over epistemic honesty. Standard RLVR paradigms, utilizing binary reward signals, inadvertently incentivize models as good test-takers rather than honest communicators, encouraging guessing whenever correctness probability exceeds zero. This paper presents an exhaustive investigation into behavioral calibration, which incentivizes models to stochastically admit uncertainty by abstaining when not confident, aligning model behavior with accuracy. Synthesizing recent advances, we propose and evaluate training interventions optimizing strictly proper scoring rules for models to output a calibrated probability of correctness. Our methods enable models to either abstain from producing a complete response or flag individual claims where uncertainty remains. Utilizing Qwen3-4B-Instruct, empirical analysis reveals behavior-calibrated reinforcement learning allows smaller models to surpass frontier models in uncertainty quantification--a transferable meta-skill decouplable from raw predictive accuracy. Trained on math reasoning tasks, our model's log-scale Accuracy-to-Hallucination Ratio gain (0.806) exceeds GPT-5's (0.207) in a challenging in-domain evaluation (BeyondAIME). Moreover, in cross-domain factual QA (SimpleQA), our 4B LLM achieves zero-shot calibration error on par with frontier models including Grok-4 and Gemini-2.5-Pro, even though its factual accuracy is much lower.

大模型幻觉强化学习不确定性量化可信AI

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。