用关键帧插值加速机器人世界模型推理,保留重要动作事件
SKIP: Sparse Keyframe Interpolation Paradigm for Efficient Embodied World Models

- 通过多模态特征识别关键动作帧,仅对关键帧生成并插值填补中间帧
- 在LIBERO上推理速度提升4.16倍,视觉质量更高且FVD降低89%
- 适合需要高效仿真数据的机器人强化学习研究者
具身世界模型在机器人领域前景广阔,可预测机器人动作对环境的影响。然而,像素空间中的长时序滚动推断计算成本高昂,通常需逐帧生成。若随意丢帧,会破坏任务相关事件(如接近、接触、抓取、释放)的完整性,影响下游策略性能。为此,我们提出稀疏关键帧插值范式SKIP,一种保持事件完整性的稀疏到密集生成框架。SKIP首先利用机器人感知的多模态特征识别任务相关的关键帧,再通过稀疏视频扩散模型合成这些关键帧。随后,基于动作条件的间隙预测器与插值器重建缺失区间。在LIBERO数据集上,SKIP生成密集轨迹的速度比全帧基线快4.16倍,视觉保真度更高,聚合FVD降低89.0%。更重要的是,SKIP生成的视频可有效用于策略训练;即使完全替代真实示范,模拟环境成功率仅下降1.3个百分点,真实机器人下降6.7个百分点,而全帧逐帧生成则崩溃下降48至58个百分点。
原文摘要 · Abstract (English)
Embodied world models have emerged as a promising paradigm in robotics by predicting how robot actions affect the surrounding scene. However, the rollout inference remains computationally expensive in pixel space, as long-horizon manipulation videos typically have to be generated frame by frame. This cost cannot be easily reduced by indiscriminately dropping frames, since downstream policies rely on complete preservation of sparse task-relevant events such as approach, contact, grasp, and release. To address this challenge, we propose Sparse Keyframe Interpolation Paradigm (SKIP), an event-preserving sparse-to-dense framework that avoids dense frame-by-frame generation. SKIP first identifies task-relevant keyframes by leveraging robot-aware multimodal features. It then synthesizes only these keyframes with a sparse video diffusion model. A learned gap predictor and an action-conditioned interpolator subsequently reconstruct the missing intervals according to the robot actions. On LIBERO, SKIP generates dense rollouts $4.16\times$ faster than a dense baseline while improving visual fidelity and reducing aggregate FVD by $89.0\%$. Importantly, SKIP-generated videos are effective policy-training data. Even when they fully replace real demonstrations, $π_{0.5}$ success drops only $1.3$ pp in LIBERO simulation and $6.7$ pp on the real robot, whereas fully dense frame-by-frame generation collapses by $48$ to $58$ pp.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。