用视觉语言模型辅助双臂协作,从少量示范快速学会复杂操作。
VLM-SFD: VLM-Assisted Siamese Flow Diffusion Framework for Dual-Arm Cooperative Manipulation
- 双编码器扩散网络将两物体映射到共享空间,生成对象中心运动流。
- 仅需少量人类示范,即可在多种任务中快速泛化并适应动态环境。
- 结合视觉语言模型动态分配最优动作,适合真实场景的双臂协作。
双臂协作操作在应对需要无缝协调与自适应动态的复杂现实任务方面具有巨大潜力。尽管基于学习的运动规划取得显著进展,但多数方法在跨不同操作任务时难以泛化,尤其在涉及两个物体交互(如装配、工具使用、双手抓取)的动态非结构化环境中表现不佳。为此,我们提出一种新颖的VLM辅助孪生流扩散框架(VLM-SFD),用于高效模仿学习。该框架通过双编码器-解码器孪生架构将两个目标物体嵌入共享潜在空间,并利用条件扩散过程(以任务指令为条件)生成双流对象中心运动流,指导双臂协同。我们进一步设计动态任务分配策略,将预测的2D运动流无缝映射至3D空间,并引入预训练视觉语言模型(VLM),实时自适应地为每只机械臂分配最优运动。实验验证了该方法的有效性,展示了其在多样化操作任务中的强大泛化能力,同时保持高效率和适应性。代码与演示视频已公开于项目网站 https://sites.google.com/view/vlm-sfd/。
原文摘要 · Abstract (English)
Dual-arm cooperative manipulation holds great promise for tackling complex real-world tasks that demand seamless coordination and adaptive dynamics. Despite substantial progress in learning-based motion planning, most approaches struggle to generalize across diverse manipulation tasks and adapt to dynamic, unstructured environments, particularly in scenarios involving interactions between two objects such as assembly, tool use, and bimanual grasping. To address these challenges, we introduce a novel VLM-Assisted Siamese Flow Diffusion (VLM-SFD) framework for efficient imitation learning in dual-arm cooperative manipulation. The proposed VLM-SFD framework exhibits outstanding adaptability, significantly enhancing the ability to rapidly adapt and generalize to diverse real-world tasks from only a minimal number of human demonstrations. Specifically, we propose a Siamese Flow Diffusion Network (SFDNet) employs a dual-encoder-decoder Siamese architecture to embed two target objects into a shared latent space, while a diffusion-based conditioning process - conditioned by task instructions - generates two-stream object-centric motion flows that guide dual-arm coordination. We further design a dynamic task assignment strategy that seamlessly maps the predicted 2D motion flows into 3D space and incorporates a pre-trained vision-language model (VLM) to adaptively assign the optimal motion to each robotic arm over time. Experiments validate the effectiveness of the proposed method, demonstrating its ability to generalize to diverse manipulation tasks while maintaining high efficiency and adaptability. The code and demo videos are publicly available on our project website https://sites.google.com/view/vlm-sfd/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。