arXiv:2510.15530cs.ROcs.CV2025-10被引 11

无需点云,仅用视觉图像实现高精度机器人抓取,性能媲美甚至超越依赖点云的方法。

VO-DP: Semantic-Geometric Adaptive Diffusion Policy for Vision-Only Robotic Manipulation

  • 利用预训练视觉模型融合语义与几何特征,通过交叉注意力实现多模态特征融合。
  • 在仿真中成功率64.6%,真实世界达87.9%,显著优于对比方法。
  • 适用于真实场景鲁棒操作,开源训练库支持多机多卡并行训练。

在模仿学习背景下,基于视觉-运动的扩散策略是机器人操作的重要方向。现有方法多依赖点云作为观测输入,并通过点云特征学习构建场景表征,从而取得优异精度。然而,对仅使用视觉输入且具有巨大潜力的方案探索仍不充分。本文提出一种仅使用单视角视觉图像的扩散策略(VO-DP),借助预训练视觉基础模型实现语义与几何特征的有效融合。具体地,采用VGGT的中间特征结合DINOv2的语义特征与交替注意力模块的几何特征,通过交叉注意力融合,并以卷积网络进行空间压缩,形成策略头输入。大量实验表明,VO-DP不仅显著优于纯视觉基线方法(DP)——仿真中平均成功率达64.6%(远超DP的34.8%),且在仿真中与点云方法DP3相当(64.0%),更在真实任务中达到87.9%,大幅领先于DP3的67.5%和DP的11.2%。进一步的鲁棒性评估显示,该方法在颜色、尺寸、背景及光照变化下均保持高度稳定。最后,我们开源了用于机器人操作的训练库,基于Accelerate构建,支持多机多卡并行与混合精度训练,兼容如DP、DP3和VO-DP等视觉-运动策略,并支持RoboTwin仿真器。

原文摘要 · Abstract (English)

In the context of imitation learning, visuomotor-based diffusion policy learning is one of the main directions in robotic manipulation. Most of these approaches rely on point clouds as observation inputs and construct scene representations through point clouds feature learning, which enables them to achieve remarkable accuracy. However, the existing literature lacks an in-depth exploration of vision-only solutions that have significant potential. In this paper, we propose a Vision-Only and single-view Diffusion Policy learning method (VO-DP) that leverages pretrained visual foundation models to achieve effective fusion of semantic and geometric features. We utilize intermediate features from VGGT incorporating semantic features from DINOv2 and geometric features from Alternating Attention blocks. Features are fused via cross-attention and spatially compressed with a CNN to form the input to the policy head. Extensive experiments demonstrate that VO-DP not only outperforms the vision-only baseline DP significantly but also exhibits distinct performance trends against the point cloud-based method DP3: in simulation tasks, VO-DP achieves an average success rate of 64.6% on par with DP3 64.0% and far higher than DP 34.8%, while in real-world tasks, it reaches 87.9%, outperforming both DP3 67.5% and DP 11.2% by a notable margin. Further robustness evaluations confirm that VO-DP remains highly stable under varying conditions including color, size, background, and lighting. Lastly, we open-source a training library for robotic manipulation. Built on Accelerate, this library supports multi-machine and multi-GPU parallel training, as well as mixed precision training. It is compatible with visuomotor policies such as DP, DP3 and VO-DP, and also supports the RoboTwin simulator.

视觉控制扩散模型机器人抓取多模态融合

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。