提升大模型事实性，减少幻觉并增强回答细节
Learning to Reason for Factuality
- 设计多维度奖励函数，兼顾事实准确、内容详尽与相关性
- 在线强化学习使模型在6个长文本评测中幻觉率下降23.1个百分点
- 适合需要高可靠性的生成任务，如学术写作与医疗咨询
推理型大语言模型（R-LLMs）虽在复杂推理任务上表现优异，但在长篇事实性评测中仍存在大量幻觉问题。现有基于FActScore等自动评估框架的离线强化学习方法难以直接用于在线强化学习，因会引发奖励劫持，导致输出更简略或无关内容。本文提出一种新型奖励函数，同时考量事实精确度、回答详尽程度和相关性，并应用在线强化学习训练高质量的事实推理模型。在六个长文本事实性基准上测试，该模型平均幻觉率降低23.1个百分点，回答详尽程度提升23%，整体帮助性无下降。
原文摘要 · Abstract (English)
Reasoning Large Language Models (R-LLMs) have significantly advanced complex reasoning tasks but often struggle with factuality, generating substantially more hallucinations than their non-reasoning counterparts on long-form factuality benchmarks. However, extending online Reinforcement Learning (RL), a key component in recent R-LLM advancements, to the long-form factuality setting poses several unique challenges due to the lack of reliable verification methods. Previous work has utilized automatic factuality evaluation frameworks such as FActScore to curate preference data in the offline RL setting, yet we find that directly leveraging such methods as the reward in online RL leads to reward hacking in multiple ways, such as producing less detailed or relevant responses. We propose a novel reward function that simultaneously considers the factual precision, response detail level, and answer relevance, and applies online RL to learn high quality factual reasoning. Evaluated on six long-form factuality benchmarks, our factual reasoning model achieves an average reduction of 23.1 percentage points in hallucination rate, a 23% increase in answer detail level, and no degradation in the overall response helpfulness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。