通过关键帧推理提升机器人世界模型的生成效率与物理合理性。
KeyWorld: Key Frame Reasoning Enables Effective and Efficient World Models

- 仅对语义关键帧计算,中间帧由轻量卷积模型插补。
- 在LIBERO上实现5.68倍加速,复杂任务生成更符合物理规律。
- 适合需要实时、高可信世界模型的机器人控制场景。
机器人世界模型虽能预测环境状态演化,但推理速度慢、生成轨迹物理合理性不足,制约实际应用。根源在于帧对帧生成存在冗余计算,且忽略关键过渡的语义重要性。为此,我们提出KeyWorld框架:先通过迭代简化运动轨迹识别关键帧,再用DiT模型从文本任务描述中推理生成物理合理的关键帧,最后由轻量级插值器填充中间帧。在LIBERO基准测试中,相比帧对帧基线,KeyWorld实现5.68倍加速,聚焦运动感知关键帧进一步提升了复杂任务下生成视频的物理真实性。该方法为实时机器人控制等需高效高可靠世界模型的领域提供了可行路径。代码已公开。
原文摘要 · Abstract (English)
Robotic world models are a promising paradigm for forecasting future environment states, yet their inference speed and the physical plausibility of generated trajectories remain critical bottlenecks, limiting their real-world applications. This stems from the redundancy of the prevailing frame-to-frame generation approach, where the model conducts costly computation on similar frames, as well as neglecting the semantic importance of key transitions. To address this inefficiency, we propose KeyWorld, a framework that improves text-conditioned robotic world models by concentrating transformers computation on a few semantic key frames while employing a lightweight convolutional model to fill the intermediate frames. Specifically, KeyWorld first identifies significant transitions by iteratively simplifying the robot's motion trajectories, obtaining the ground truth key frames. Then, a DiT model is trained to reason and generate these physically meaningful key frames from textual task descriptions. Finally, a lightweight interpolator efficiently reconstructs the full video by inpainting all intermediate frames. Evaluations on the LIBERO benchmark demonstrate that KeyWorld achieves a 5.68$\times$ acceleration compared to the frame-to-frame generation baseline, and focusing on the motion-aware key frames further contributes to the physical validity of the generated videos, especially on complex tasks. Our approach highlights a practical path toward deploying world models in real-time robotic control and other domains requiring both efficient and effective world models. Code is released at https://anonymous.4open.science/r/Keyworld-E43D.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。