用数据飞轮自进化，让大模型在稀疏奖励下高效完成长序列规划。
Beyond Policy Optimization: A Data Curation Flywheel for Sparse-Reward Long-Horizon Planning
- 三阶段框架：初始推理、分层迁移、奖励筛选自优化
- 在多个环境上达到顶尖性能，且节省大量计算token
- 适合研究智能体规划与高效推理的开发者
大型语言推理模型在静态任务中表现优异，但在交互环境中进行多轮智能体规划时面临两大挑战：一是稀疏奖励下信用分配困难，传统强化学习失效；二是详细推理过程带来巨大计算开销。为此，我们提出BPO框架，包含三个阶段（启动、外推、精炼），构建自增强的数据飞轮，以训练适用于长周期、稀疏奖励环境的鲁棒推理模型。首先，通过规划四元数结合长短思维链实现高效推理；其次，采用复杂度分层课程学习将模型泛化至分布外任务；最后，模型通过奖励引导的拒绝采样仅学习高价值经验，实现自我迭代优化。在ALFWorld、ScienceWorld和WebShop上的实验表明，该方法达到当前最优水平，并具备显著的令牌效率优势，为智能体规划中的推理模型提供了新范式。
原文摘要 · Abstract (English)
Large Language Reasoning Models have demonstrated remarkable success on static tasks, yet their application to multi-round agentic planning in interactive environments faces two fundamental challenges. First, the intractable credit assignment problem renders conventional reinforcement learning ineffective in sparse-reward settings. Second, the computational overhead of verbose, step-by-step reasoning histories is prohibitive. To address these challenges, we propose BPO, a three-stage framework (bootstrapping, extrapolation, and refinement) that establishes a self-improving data flywheel to develop robust reasoning models for long-horizon, sparse-reward environments. Our framework first bootstraps efficient reasoning using the proposed planning quaternions with long-short chain-of-thought fusion. It then extrapolates to out-of-distribution tasks through complexity-stratified curriculum learning. Finally, the model iteratively refines itself by learning exclusively on experiences selected via reward-gated rejection sampling. Experiments on ALFWorld, ScienceWorld, and WebShop demonstrate that our approach achieves state-of-the-art with significant token efficiency, providing a new recipe for reasoning models in agentic planning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。