让机器像人一样思考：用神经网络自动优化复杂任务的计划。
Reinforced Planning with Latent World Models
- 用强化学习训练神经网络，自动改进多步动作规划。
- 在多个任务中接近完美成功率，推理速度比最强方法快67倍。
- 可独立于世界模型使用，只需1000倍更少的模拟次数。
人类通过构建计划并用内部世界模型模拟结果来解决复杂问题。机器学习已生成能预测动作序列结果的世界模型，但候选计划的改进仍未能充分学习。现有规划器要么手工设计，要么从手工优化器中提炼，或仅用于辅助策略学习而非修改计划本身。我们提出强化规划（Reinforced Planning），其核心思想是通过强化好的搜索规则，让神经规划器学会搜索。我们的实现RP1通过离线训练的优化器，同时学习评估想象结果（由评判器完成）和改进多步计划，完全基于预训练世界模型的想象推演。据我们所知，这是首个完全学习如何改进多步计划的方法。此外，它可独立于任何预训练潜空间世界模型使用。在视觉导航、机械臂抓取及机器人操作任务中，使用两种世界模型骨干网络，RP1显著优于手工设计的搜索算法，在多个场景达到近完美成功率，同时仅需1000倍更少的世界模型推演，且并发推理下最快可达67倍加速。
原文摘要 · Abstract (English)
Humans solve complex problems by constructing plans and mentally simulating their outcomes with an internal model of the world. Machine learning has produced world models that similarly predict the outcomes of action sequences, but the improvement of candidate plans still isn't fully learned. Current planners are either hand-designed, distilled from a hand-designed optimizer, or learned only to inform an amortized policy rather than to revise the plan itself. We introduce the Reinforced Planning, a method based on the idea that search can be learned by reinforcing good search rules into a neural planner. Our implementation RP1 learns both how to evaluate imagined outcomes through a critic, as well as how to improve multi-step plans through an optimizer trained fully offline from imagined world-model roll-outs. To our knowledge, RP1 is the first method to fully learn how to improve multi-step plans. Furthermore, it can be trained independently of and attached to any pretrained latent world model. Across visual navigation, arm reaching, and robotic manipulation on two world-model backbones, RP1 substantially outperforms hand-designed search algorithms, reaching near-perfect success in several settings while using $1,000 \times$ less world-model rollouts and being up to $67 \times$ faster than the strongest alternative under concurrent planner inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。