arXiv:2602.23814cs.CV2026-02中稿 · CVPR被引 5

用RGB图直接预测双手操作的3D场景变化,提升机器人空间理解能力。

Action-Geometry Prediction with 3D Geometric Prior for Bimanual Manipulation

  • 基于预训练3D几何模型,融合2D语义与本体感知构建统一状态表征。
  • 联合预测动作序列与未来3D点云,实现高精度空间演化预测。
  • 仅需RGB输入即可达成顶尖操作成功率,适合真实场景部署。

双手操作需要策略能够理解三维几何结构,预判其在动作作用下的演变,并生成流畅协调的动作。然而现有方法多依赖2D特征,空间感知有限;或需难以可靠获取的显式点云。近期3D几何基础模型表明,可从RGB图像快速鲁棒地重建精确多样的3D结构。本文利用此机会,提出一种直接基于预训练3D几何基础模型的双手操作框架。该策略将几何感知隐变量、2D语义特征与本体感知融合为统一状态表示,采用扩散模型联合预测未来动作块与未来3D隐变量,后者解码为稠密点云。通过显式预测场景演化与动作序列的协同关系,策略仅凭RGB观测即具备强空间理解与预测能力。我们在RoboTwin仿真基准及真实机器人上评估该方法,性能持续优于基于2D和点云的基线,在操作成功率、双臂协调性与3D空间预测精度上达到当前最优。代码已开源:https://github.com/Chongyang-99/GAP.git。

原文摘要 · Abstract (English)

Bimanual manipulation requires policies that can reason about 3D geometry, anticipate how it evolves under action, and generate smooth, coordinated motions. However, existing methods typically rely on 2D features with limited spatial awareness, or require explicit point clouds that are difficult to obtain reliably in real-world settings. At the same time, recent 3D geometric foundation models show that accurate and diverse 3D structure can be reconstructed directly from RGB images in a fast and robust manner. We leverage this opportunity and propose a framework that builds bimanual manipulation directly on a pre-trained 3D geometric foundation model. Our policy fuses geometry-aware latents, 2D semantic features, and proprioception into a unified state representation, and uses diffusion model to jointly predict a future action chunk and a future 3D latent that decodes into a dense pointmap. By explicitly predicting how the 3D scene will evolve together with the action sequence, the policy gains strong spatial understanding and predictive capability using only RGB observations. We evaluate our method both in simulation on the RoboTwin benchmark and in real-world robot executions. Our approach consistently outperforms 2D-based and point-cloud-based baselines, achieving state-of-the-art performance in manipulation success, inter-arm coordination, and 3D spatial prediction accuracy. Code is available at https://github.com/Chongyang-99/GAP.git.

双手操作3D几何扩散模型视觉导航

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。