arXiv:2607.06223cs.AI2026-07被引 2

用信息增益动态分配计算资源,让大模型在多轮推理中更聪明地探索。

Information Gain-based Rollout Policy Optimization: An Adaptive Tree-Structured Rollout Approach for Multi-Turn LLM Agents

论文配图:Information Gain-based Rollout Policy Optimization: An Adaptive Tree-Structured Rollout Approach for Multi-Turn LLM Agents
图 1 · 摘自论文原文
  • 根据中间状态的信息量分配计算预算,优先拓展有价值分支。
  • 在7个问答任务上,相同预算下效果优于现有方法。
  • 适合需要长序列决策的复杂推理场景,如智能客服、科研助手。

强化学习已成为提升大语言模型在长程搜索任务中表现的有前景范式,其中代理需在获得最终结果前做出一系列中间决策。然而,现有方法存在关键局限:滚动(rollout)预算常未考虑中间状态的价值,导致大量计算浪费在低信息量的状态上。本文提出基于信息增益的滚动策略优化(IGRPO),将中间状态的信息量作为滚动收集的组织原则。具体而言,IGRPO通过预算感知的树状结构滚动,依据节点级信息量分配扩展预算,使高信息量分支获得更多扩展机会,同时逐步抑制无望分支。我们进一步证明,基于信息增益的滚动会诱导出轨迹上的显式教师分布,自然形成清晰的策略优化目标,从而在单一框架内统一自适应树状探索与严谨的策略学习。在七个具有挑战性的搜索增强型问答基准上的实验表明,IGRPO在相同滚动预算约束下持续优于强基线,验证了利用诱导教师分布指导策略优化的有效性。

原文摘要 · Abstract (English)

Reinforcement learning has become a promising paradigm for improving large language model (LLM) agents on long-horizon search tasks, where the agent must make a sequence of intermediate decisions before receiving a final outcome. However, existing methods still face a key limitation: the rollout budget is often allocated without explicitly assessing the utility of intermediate states. As a result, substantial computation may be spent on low-value states, even though different branches can vary drastically in their informativeness. In this paper, we propose Information Gain-based Rollout Policy Optimization (IGRPO), a policy optimization framework that treats intermediate-state informativeness as the organizing principle of rollout collection. Specifically, IGRPO performs budget-aware tree-structured rollouts by allocating expansion budget according to node-level informativeness, so that more informative branches are expanded more frequently while unpromising branches are progressively suppressed. We further demonstrate that the information gain-based rollout induces an explicit limiting teacher distribution over trajectories, which naturally yields a clear policy optimization target, thereby unifying adaptive tree-structured exploration with principled policy learning under a single framework. Experiments on seven challenging search-augmented QA benchmarks demonstrate that IGRPO consistently outperforms strong baselines under the same rollout budget constraints, validating the effectiveness of leveraging the induced teacher distribution to guide policy optimization for long-horizon search agents.

强化学习大模型推理自适应探索

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。