用可验证奖励提升视频推理模型的规划能力
Wan-R1: Verifiable-Reinforcement Learning for Video Reasoning
- 设计基于任务指标的可验证奖励函数，替代易失效的多模态奖励
- 在3D迷宫中准确率提升29.1%，陷阱规避任务提升51.4%
- 适合研究视频生成、强化学习与智能体规划的开发者
视频生成模型虽能生成视觉连贯内容，但在需要空间推理和多步规划的任务上表现不佳。强化学习（RL）有望提升泛化能力，但其效果高度依赖奖励设计——这一关键问题尚未系统研究。本文将分组相对策略优化（GRPO）适配到流模型视频生成框架，在迷宫求解与机器人导航任务中训练。实验发现，多模态奖励模型在此类任务中会灾难性失败。为此，我们设计基于客观任务指标的可验证奖励：针对结构化游戏环境引入多组件轨迹奖励，针对机器人导航提出嵌入级可验证奖励。结果表明，使用可验证奖励进行强化学习微调显著提升泛化性能。例如，在复杂3D迷宫中，模型精确匹配准确率比SFT基线提升29.1%；在避障任务中提升51.4%。系统性分析揭示，可验证奖励对训练稳定性至关重要，而多模态奖励可能导致退化解。这些发现确立可验证奖励设计为鲁棒视频推理的关键支撑。代码将公开。
原文摘要 · Abstract (English)
Video generation models produce visually coherent content but struggle with tasks requiring spatial reasoning and multi-step planning. Reinforcement learning (RL) offers a path to improve generalization, but its effectiveness in video reasoning hinges on reward design -- a challenge that has received little systematic study. We investigate this problem by adapting Group Relative Policy Optimization (GRPO) to flow-based video models and training them on maze-solving and robotic navigation tasks. We first show that multimodal reward models fail catastrophically in this setting. To address this, we design verifiable reward functions grounded in objective task metrics. For structured game environments, we introduce a multi-component trajectory reward. For robotic navigation, we propose an embedding-level verifiable reward. Our experiments show that RL fine-tuning with verifiable rewards improves generalization. For example, on complex 3D mazes, our model improves exact match accuracy by 29.1\% over the SFT baseline, and on trap-avoidance tasks by 51.4\%. Our systematic reward analysis reveals that verifiable rewards are critical for stable training, while multimodal reward models could lead to degenerate solutions. These findings establish verifiable reward design as a key enabler for robust video reasoning. Code will be publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。