arXiv:2511.09109cs.CLcs.AI2025-11AAAI

双向评估推理步骤,提升大模型复杂问题回答准确性。

Thinking Forward and Backward: Multi-Objective Reinforcement Learning for Retrieval-Augmented Reasoning

  • 通过正反向联合评估每步推理完整性,避免中间步骤偏差。
  • 在7个问答数据集上超越现有方法,显著减少幻觉现象。
  • 适合需要可靠多步推理的智能客服、学术研究等场景。

检索增强生成(RAG)在缓解大语言模型幻觉方面已证明有效,但在复杂的多步推理任务中表现仍受限。近期工作将基于搜索的交互引入RAG,实现与实时检索结合的迭代推理。然而,多数方法依赖结果导向的监督,对中间步骤缺乏显式指导,常导致奖励欺骗和响应质量下降。本文提出Bi-RAR,一种新型检索增强推理框架,通过正向与反向联合评估每个中间步骤。为衡量每步的信息完备性,引入基于柯尔莫哥洛夫复杂性的双向信息距离,以语言模型生成概率近似计算。该度量同时反映当前推理距离答案的远近以及对问题的覆盖程度。为在双向信号下优化推理,采用具有级联奖励结构的多目标强化学习框架,强调早期轨迹对齐。实验证明,Bi-RAR在七个问答基准上均优于先前方法,支持训练与推理阶段高效地与搜索引擎互动。

原文摘要 · Abstract (English)

Retrieval-augmented generation (RAG) has proven to be effective in mitigating hallucinations in large language models, yet its effectiveness remains limited in complex, multi-step reasoning scenarios. Recent efforts have incorporated search-based interactions into RAG, enabling iterative reasoning with real-time retrieval. Most approaches rely on outcome-based supervision, offering no explicit guidance for intermediate steps. This often leads to reward hacking and degraded response quality. We propose Bi-RAR, a novel retrieval-augmented reasoning framework that evaluates each intermediate step jointly in both forward and backward directions. To assess the information completeness of each step, we introduce a bidirectional information distance grounded in Kolmogorov complexity, approximated via language model generation probabilities. This quantification measures both how far the current reasoning is from the answer and how well it addresses the question. To optimize reasoning under these bidirectional signals, we adopt a multi-objective reinforcement learning framework with a cascading reward structure that emphasizes early trajectory alignment. Empirical results on seven question answering benchmarks demonstrate that Bi-RAR surpasses previous methods and enables efficient interaction and reasoning with the search engine during training and inference.

推理增强强化学习多步推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。