让大模型像研究者一样分步规划，提升长文本回答质量。
Planner-Centric Reinforcement Learning for Deep Research with Structure-Aware Reward
- 用有向无环图显式表示研究计划，拆解任务并结构化执行。
- 在多个长文本评测上超越基线5.1至8.0分，提升明显。
- 适合需要复杂推理与多分支探究的学术级问答场景。
深度研究任务要求大模型规划研究方向、检索证据并跨多分支综合生成长篇答案。现有训练范式或依赖短答案可验证的问答作为代理，或优化单一长轨迹，导致规划与执行难以分离，且规划过程信用分配不足。本文提出DecomposeR，一种以规划为中心的深度研究框架，将研究计划表示为带类型的有向无环图（DAG），使规划过程显式、结构化且可奖励。我们使用Qwen3-8B模型分两阶段训练：首先通过规划器强化学习（Planner RL）学习图结构与查询分解以优化规划能力；随后通过回答器强化学习（Answerer RL）在已学计划基础上学习分支执行与最终合成。通过奖励显式规划标记与结构化组件而非扁平轨迹，DecomposeR实现更细粒度的规划优化，降低端到端训练的歧义性。实验表明，DecomposeR-8B在多个主流长文本基准上相较强基线提升5.1至8.0分，得益于更优的规划与回答能力。
原文摘要 · Abstract (English)
Deep research tasks require LLMs to plan what to investigate, retrieve evidence, and synthesize long-form answers across multiple branches of inquiry. Existing training paradigms either rely on short-form verifiable QA as a proxy or optimize monolithic long trajectories, which makes planning and execution difficult to disentangle and yields weak credit assignment for the planning process. We propose DecomposeR, a planner-centric deep research framework that represents research plans as typed directed acyclic graphs (DAGs), allowing planning to be made explicit, structured, and rewardable. We train a Qwen3-8B model in two stages: planner reinforcement learning (RL) first learns graph structure and query decomposition to improve research planning, and answerer reinforcement learning (RL) then learns branch-level execution and final synthesis conditioned on the learned plan. By assigning rewards to explicit planner tokens and structured components rather than to a flat trajectory, DecomposeR enables finer-grained optimization of planning while reducing the ambiguity of end-to-end training. Experiments show that DecomposeR-8B improves over strong comparable open baselines by 5.1-8.0 points on popular long-form benchmarks due to improved planning and answering capabilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。