通过可逆动作循环实现视频世界模型的自验证强化学习,减少长期误差。
WorldCycle: Self-Verifiable Reinforcement Learning for Long-Horizon Video World Models

- 利用动作循环的可逆性生成无监督的长期状态验证信号。
- 在复杂动作组合下,状态回归误差降低44%,复合动作准确率提升近4倍。
- 适合需要高精度长时序预测的机器人规划与物理模拟任务。
交互式视频世界模型对长时程规划与探索至关重要,但存在误差累积问题。后训练强化学习(RL)可改善模型性能,却面临验证瓶颈:任意动作序列缺乏真实未来状态以衡量长期漂移。本文关键洞察是:可逆动作循环可实现验证——正向与反向动作序列组合应严格返回初始状态,从而提供无标注的长期正确性监督。基于此,我们提出WorldCycle框架,从普通动作序列构建闭合动作循环并重复执行,优化两类互补奖励:空间闭合奖励(强制镜像前后段对称),时间一致性奖励(对齐多次循环执行的状态)。这两类奖励促使模型将动作视为一致的状态算子而非记忆化的时间模式,并自然扩展至基模型处理不佳的分布外复合动作循环。我们还发布了CycleBench,用于诊断复杂动作结构下的状态返回能力。实验表明,WorldCycle将状态回归漂移降低最多44%,复合动作准确率提升近4倍,为物理可信的世界模型提供了关键基础。
原文摘要 · Abstract (English)
Interactive video world models are essential for long-horizon planning and exploration, yet they suffer from compounding errors. Post-training methods such as reinforcement learning (RL) can improve these models, but they hit a verification bottleneck: for arbitrary action sequences, no ground-truth future state exists to measure long-term drift. Our key insight is that reversible action cycles make this verification possible: a sequence composed with its inverse must analytically return to the initial state, yielding annotation-free supervision on long-horizon correctness. Building on this, we introduce WorldCycle, a self-verifiable RL framework that constructs closed action cycles and their repeated executions from ordinary action sequences, and optimizes two complementary rewards: a spatial closure reward enforcing symmetry between mirrored forward and reverse segments, and a temporal consistency reward aligning states across repeated cycle executions. These rewards force the model to learn actions as consistent state operators rather than memorized temporal patterns, and extend naturally to out-of-distribution composite action cycles that the base model handles poorly. We further release CycleBench, a diagnostic benchmark for state-returning ability under complex action structures. WorldCycle reduces state returning drift by up to 44% and lifts composite-action accuracy nearly 4x over the base model, providing a vital foundation for physically grounded world models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。