通过平衡的词元级奖励,让大模型生成更可信且不冗长的回答。
BALTO: Balanced Token-Level Policy Optimization for Hallucination Mitigation

- 将事实断言转化为词元级标签,实现精准反馈
- 在六个评测场景中,信仰度最高,优于现有方法
- 适合需要高可信度的问答与检索增强应用
幻觉仍是大语言模型在知识密集型场景中部署的主要障碍,生成内容需忠实于提供证据。强化学习是缓解幻觉的有前景方向,但响应级奖励存在粒度不匹配问题:局部幻觉会导致支持内容被错误惩罚。尽管已有工作引入声明级验证和词元级奖励,不平衡的信用分配仍可能引发长度、冗余或优化噪声偏差。我们提出BALTO,一种用于幻觉缓解的平衡词元级策略优化框架。BALTO提取可验证的事实断言,对照参考上下文进行验证,并将断言级判断投影为词元级标签。框架引入平衡的词元级信用分配机制,将未支持内容的概率质量重新分配给忠实内容,而非整体抑制响应。我们从理论角度系统分析了响应级奖励的局限性,并证明BALTO在训练稳定性和优化效率上的优势。在ConFiQA、RAGTruth和FinLLM-Eval上的实验表明,BALTO在所有六个模型-基准组合中均达到最高信仰度,且在Q-Score上持续优于现有后训练基线,展现出更强的信仰度-信息量权衡能力。
原文摘要 · Abstract (English)
Hallucinations remain a major obstacle to deploying large language models (LLMs) in knowledge-intensive settings, where generated responses must be faithfully grounded in provided evidence. Reinforcement learning (RL) is a promising direction for hallucination mitigation, but response-level faithfulness rewards suffer from a granularity mismatch: localized hallucinations can cause supported content to receive spurious penalties. Although recent work introduces fine-grained feedback such as claim-level verification and token-level rewards, unbalanced credit assignment can still induce length, verbosity, or optimization-noise biases. We propose BALTO, a Balanced Token-level Policy Optimization framework for hallucination mitigation. BALTO extracts checkable factual claims, verifies them against the reference context, and projects claim-level judgments to token-level labels. A balanced token-level credit assignment mechanism is introduced into the framework. This design redistributes probability mass from unsupported content toward faithful content, rather than suppressing the entire response. We systematically analyze the limitations of response-level rewards from a theoretical standpoint, and prove BALTO's advantages in training stability and optimization efficiency for hallucination mitigation. Experiments on ConFiQA, RAGTruth, and FinLLM-Eval show that BALTO achieves the highest faithfulness across all six model--benchmark settings and consistently outperforms existing post-training baselines in Q-Score, demonstrating a stronger faithfulness--informativeness trade-off.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。