用统一去噪过程同时预测图像和机器人动作，提升控制泛化能力。
Prediction with Action: Visual Policy Learning via Joint Denoising Process
- 通过扩散变换器联合建模图像与机器人状态，实现图像与动作同步去噪。
- 在Metaworld上相对基线提升26.3%，真实场景任务成功率提高28.0%。
- 支持多模态输入与跨数据集协同训练，适合数据效率要求高的机器人学习。
扩散模型在图像生成任务中表现出色，如图像编辑与视频生成，展现出对物理世界的良好理解。与此同时，扩散模型在机器人控制任务中也展现出潜力，通过去噪动作实现扩散策略（diffusion policy）。尽管生成模型与扩散策略分别擅长图像预测与动作生成，但二者技术上均遵循相似的去噪流程。在机器人任务中，未来图像预测与动作生成密切相关，因它们共享相同的物理世界动态。基于此，我们提出PAD框架，将图像预测与机器人动作生成统一于联合去噪过程中。PAD采用扩散变换器（DiT）无缝融合图像与机器人状态，实现未来图像与动作的同步预测。此外，PAD支持在机器人示范数据与大规模视频数据集上协同训练，并可轻松扩展至深度图像等其他模态。在仅使用单个文本条件视觉策略的数据高效模仿学习设置下，PAD在完整Metaworld基准测试中取得26.3%的相对性能提升。此外，在真实机器人操作任务中，其对未见任务的泛化能力显著优于最强基线，成功率提升28.0%。
原文摘要 · Abstract (English)
Diffusion models have demonstrated remarkable capabilities in image generation tasks, including image editing and video creation, representing a good understanding of the physical world. On the other line, diffusion models have also shown promise in robotic control tasks by denoising actions, known as diffusion policy. Although the diffusion generative model and diffusion policy exhibit distinct capabilities--image prediction and robotic action, respectively--they technically follow a similar denoising process. In robotic tasks, the ability to predict future images and generate actions is highly correlated since they share the same underlying dynamics of the physical world. Building on this insight, we introduce PAD, a novel visual policy learning framework that unifies image Prediction and robot Action within a joint Denoising process. Specifically, PAD utilizes Diffusion Transformers (DiT) to seamlessly integrate images and robot states, enabling the simultaneous prediction of future images and robot actions. Additionally, PAD supports co-training on both robotic demonstrations and large-scale video datasets and can be easily extended to other robotic modalities, such as depth images. PAD outperforms previous methods, achieving a significant 26.3% relative improvement on the full Metaworld benchmark, by utilizing a single text-conditioned visual policy within a data-efficient imitation learning setting. Furthermore, PAD demonstrates superior generalization to unseen tasks in real-world robot manipulation settings with 28.0% success rate increase compared to the strongest baseline. Project page at https://sites.google.com/view/pad-paper
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。