用逻辑约束提升视频生成的物理合理性,让机器人操作更可靠。
CreFlow: Corrective Reflow for Sparse-Reward Embodied Video Diffusion RL

- 将任务要求转为逻辑公式,生成时自动验证是否符合
- 训练后成功率提升23.8个百分点,跨八类双臂任务有效
- 适合需要高精度动作生成的机器人控制研究者
在异构数据上训练的视频生成模型虽能生成视觉逼真的操作视频,但常违背物理规律。现有强化学习奖励仅依赖低层视觉指标,难以评估复杂任务。本文提出基于线性时序逻辑(LTL)的约束奖励模型,自动将任务规范转化为可组合的逻辑约束,提供准确奖励与错误定位。为此设计了CreFlow框架:一是信用感知的NFT损失,仅更新与奖励相关的区域;二是校正流损失,利用组内正样本估计修正方向,加速并稳定训练。实验表明,该方法在奖励判断上更贴近人类与仿真成功标签,且在八种双臂操作任务中提升下游执行成功率23.8个百分点。
原文摘要 · Abstract (English)
Video generation models trained on heterogeneous data with likelihood-surrogate objectives can produce visually plausible rollouts that violate physical constraints in embodied manipulation. Although reinforcement-learning post-training offers a natural route to adapting VGMs, existing video-RL rewards often reduce each rollout to a low-level visual metric, whereas manipulation video evaluation requires logic-based verification of whether the rollout satisfies a compositional task specification. To fill this gap, we introduce a compositional constraint-based reward model for post-training embodied video generation models, which automatically formulates task requirements as a composition of Linear Temporal Logic constraints, providing faithful rewards and localized error information in generated videos. To achieve effective improvement in high-dimensional video generation using these reward signals, we further propose CreFlow, a novel online RL framework with two key designs: i) a credit-aware NFT loss that confines the RL update to reward-relevant regions, preventing perturbations to unrelated regions during post-training; and ii) a corrective reflow loss that leverages within-group positive samples as an explicit estimate of the correction direction, stabilizing and accelerating training. Experiments show that CreFlow yields reward judgments better aligned with human and simulator success labels than existing methods and improves downstream execution success by 23.8 percentage points across eight bimanual manipulation tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。