arXiv:2505.20825cs.CL2025-05ACL被引 4

提出可验证的增强型信息优化框架,提升长文本生成的准确性和稳定性。

Reinforced Informativeness Optimization for Long-Form Retrieval-Augmented Generation

  • 以可度量的信息性为强化学习目标,实现外部可验证的奖励设计。
  • 通过跨源核验机制,显著提升事实召回率与忠实度,优于现有方法。
  • 无需人工标注或强教师模型,适合追求可信长文本生成的研究者。

长文本问答(LFQA)需要从多源证据中合成连贯且事实正确的长篇回答,这对强化学习(RL)的奖励设计提出更高要求。现有奖励常依赖唯一答案与精确匹配,适用于短答案或数学题,但在长文本场景下失效,导致反馈信号不稳定、优化效果不佳。本文提出RioRAG框架,首次将信息性定义为可测量、可外部验证的强化学习目标。通过基于要点的跨源验证机制,支持小规模大模型自我演进,并提供更密集、更具区分度的奖励信号,缓解奖励稀疏问题,实现稳定优化。该方法不依赖人工监督或强教师模型,仅使用可验证反馈。在LongFact和RAGChecker数据集上的实验表明,RioRAG显著提升事实召回率与忠实度,确立可验证奖励建模作为可信长文本RAG的基础。代码已开源:https://github.com/RUCAIBox/RioRAG。

原文摘要 · Abstract (English)

Long-form question answering (LFQA) requires open-ended long-form responses that synthesize coherent, factually grounded content from multi-source evidence. This makes reinforcement learning (RL) reward design critical. The reward must be verifiable for faithful grounding and stable optimization. However, many standard rewards assume a unique target with an exact-match notion of correctness, which fits short-form QA and math but breaks in LFQA. As a result, current RAG systems still lack verifiable reward mechanisms, yielding unstable feedback signals and suboptimal optimization outcomes. We propose RioRAG, a framework for reinforced verifiable informativeness optimization. First, it defines informativeness as a measurable and externally verifiable objective for RL. Second, RioRAG uses nugget-centric verification with cross-source checks to enable self-evolution of smaller LLMs and to provide denser, action-discriminative rewards that mitigate reward sparsity and stabilize optimization. This formulation avoids handcrafted supervision for the policy model and strong teacher-model distillation, relying instead on externally verifiable feedback. Experiments on LongFact and RAGChecker show that RioRAG achieves higher factual recall and faithfulness, establishing verifiable reward modeling as a foundation for trustworthy long-form RAG. Our codes are available at https://github.com/RUCAIBox/RioRAG.

长文本生成强化学习可信生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。