让机器人自己搞定摄像头位置,不用校准就能适应不同视角。
From Fixed to Free Cameras: Calibration-Free View-Robust Vision-Language-Action Model

- 用相机坐标系预测动作,再自动计算到机器人基座的变换
- 在多种未见过的视角下,成功率显著高于传统方法
- 适合部署在摄像头常移动的真实场景,无需额外校准
真实世界机器人部署中,摄像头位置往往与训练阶段不同,常见重新安装或调整。现有视图鲁棒的视觉-语言-动作(VLA)策略仅在已知相机外参时有效,导致使用脆弱且难以部署。本文提出相机中心的VLA模型CamVLA,通过预测(i)以相机坐标系表示的末端执行器动作,以及(ii)连接相机与机器人基座的6-DoF手眼矩阵,实现动作与相机几何的解耦。通过确定性几何变换组合两个预测,生成机器人基座帧下的动作。该策略无需相机标定、无需深度信息、仅需单张单目RGB图像和任务指令即可运行。仿真与真实机器人实验均表明,CamVLA在多种未见视角下持续提升任务成功率。项目页面:https://alibaba-damo-academy.github.io/CamVLA/
原文摘要 · Abstract (English)
Real-world robot deployment rarely maintains the training-stage camera setup, where cameras often experience repositioning or remounting depending on actual scenarios. Existing view-robust Vision-Language-Action (VLA) policies tolerate such camera variations only when the camera extrinsics are explicitly provided, making them fragile and hard to use especially when view robustness is critical. We argue that the policy should not be told where the camera is, but rather figure it out by itself. To this end, we introduce Camera-Centric VLA (CamVLA), a new VLA model that decouples manipulation controls from camera geometry by predicting (i) a camera-centric end-effector action expressed in the local camera frame, and (ii) a 6-DoF hand-eye matrix relating cameras to the robot base. A deterministic geometric transformation composes the two predictions into a robot base-frame action. This disentangles how I should move in pose-independent camera-centric action generation from where I am looking from in camera-perspective geometric grounding. The resulting policy is calibration-free, depth-free, and single-view, requiring only a single monocular RGB image as the visual observation and task instruction at deployment. Evaluations in both simulation and real-world robot data show that CamVLA consistently improves success rates across diverse unseen viewpoints. Project page: https://alibaba-damo-academy.github.io/CamVLA/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。