用GAN与扩散模型无监督替换视频中的机械臂,提升机器人跨平台学习数据质量。
RoboSwap: A GAN-driven Video Diffusion Framework For Unsupervised Robot Arm Swapping
- 分离机械臂与背景,用无配对GAN实现跨臂迁移。
- 融合扩散模型优化运动真实感与交互一致性,结构一致率超基线12%。
- 适合缺乏配对数据的机器人跨体感学习场景,尤其适用于多环境泛化。
生成模型的进步推动了视频合成与编辑的发展,但高质量、多样化的数据集仍然稀缺,限制了视频条件下的机器人学习及跨平台泛化能力。本文提出RoboSwap框架,解决将一段视频中的机械臂替换成另一机械臂的问题,这是实现跨体感学习的关键步骤。不同于依赖相同环境配对视频演示的先前方法,RoboSwap在来自不同环境的无配对数据上运行,显著降低数据收集成本。该框架整合了GAN与扩散模型,构建新颖的视频编辑流程:首先分割机械臂与背景,训练无配对GAN实现机械臂间转换;再将转换后的机械臂与原始视频背景融合,并通过扩散模型进行精细化处理,提升结构连贯性、运动真实性和物体交互自然度。两个阶段独立训练。实验表明,RoboSwap在三个基准测试中均优于当前最优的视频与图像编辑模型,在结构一致性和运动一致性方面表现更优,为机器人学习提供了可靠的跨体感数据生成方案。
原文摘要 · Abstract (English)
Recent advancements in generative models have revolutionized video synthesis and editing. However, the scarcity of diverse, high-quality datasets continues to hinder video-conditioned robotic learning, limiting cross-platform generalization. In this work, we address the challenge of swapping a robotic arm in one video with another: a key step for crossembodiment learning. Unlike previous methods that depend on paired video demonstrations in the same environmental settings, our proposed framework, RoboSwap, operates on unpaired data from diverse environments, alleviating the data collection needs. RoboSwap introduces a novel video editing pipeline integrating both GANs and diffusion models, combining their isolated advantages. Specifically, we segment robotic arms from their backgrounds and train an unpaired GAN model to translate one robotic arm to another. The translated arm is blended with the original video background and refined with a diffusion model to enhance coherence, motion realism and object interaction. The GAN and diffusion stages are trained independently. Our experiments demonstrate that RoboSwap outperforms state-of-the-art video and image editing models on three benchmarks in terms of both structural coherence and motion consistency, thereby offering a robust solution for generating reliable, cross-embodiment data in robotic learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。