用双目视觉提升机器人抓取的三维感知能力
StereoPolicy: Improving Robotic Manipulation Policies via Stereo Perception
- 通过双目图像对直接融合左右视图特征，隐式获取深度信息
- 在多个仿真与真实机器人任务中性能优于单目、深度图等基线方法
- 适合需要精准三维操作的机器人抓取场景，如复杂桌面环境
近期机器人模仿学习已能基于视觉输入操控多种物体。然而，单目观测缺乏深度信息，在杂乱或几何复杂的场景中难以实现精确操作。显式的深度图和点云在真实场景中常噪声大且不稳定。我们提出 StereoPolicy，一种直接利用同步双目图像对提升几何推理能力的视觉-运动策略框架，无需构建显式三维表示。该框架使用预训练的二维视觉编码器处理每张图像，并通过基于交叉注意力的双目变换器融合左右特征，隐式捕捉空间对应关系与视差线索。StereoPolicy 可集成于基于扩散模型和预训练视觉-语言-动作（VLA）的策略中，在三个仿真基准和七个真实机器人桌面及双手移动操作任务中，持续优于 RGB、RGB-D、点云和多视角基线方法。结果表明，双目视觉有效连接了二维预训练表征与三维几何理解，助力机器人操作。
原文摘要 · Abstract (English)
Recent advances in robot imitation learning have produced powerful visuomotor policies that manipulate diverse objects from visual inputs. However, monocular observations lack depth information, which is critical for precise manipulation in cluttered or geometrically complex scenes. Explicit depth maps and point clouds are often noisy and fragile in real-world manipulation. We introduce StereoPolicy, a visuomotor policy learning framework that directly leverages synchronized stereo image pairs to improve geometric reasoning without constructing explicit 3D representations. StereoPolicy processes each image with pretrained 2D vision encoders and fuses left-right features through a cross-attention-based Stereo Transformer, capturing spatial correspondence and disparity cues implicitly. The framework integrates with diffusion-based and pretrained vision-language-action (VLA) policies, delivering consistent improvements over RGB, RGB-D, point cloud, and multi-view baselines across three simulation benchmarks and seven real-robot tabletop and bimanual mobile manipulation tasks. Our results show that stereo vision bridges 2D pretrained representations and 3D geometric understanding for robotic manipulation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。