让机器人主动调整摄像头视角，提升视觉反馈精度。
Active Vision Might Be All You Need: Exploring Active Vision in Bimanual Robotic Manipulation
- 通过人类示范学习动态调整相机视角的策略。
- 在视线受限任务中，性能显著优于固定摄像头。
- 适用于需要精细视觉引导的双臂操作场景。
模仿学习在利用视觉反馈执行高精度操作任务方面展现出巨大潜力，但通常采用固定位置的摄像头，导致遮挡和视野受限。摄像头常置于通用位置，缺乏针对具体任务的最佳视角。本文研究主动视觉（AV）在模仿学习与操作中的作用，提出一种新方法：除操作策略外，机器人还从人类示范中学习主动视觉策略，动态调整摄像头视角以获取更优环境信息。我们构建了基于ALOHA 2的双臂遥操作系统AV-ALOHA，新增一个仅携带立体摄像头的7-DoF机械臂，专门负责寻找最佳视角。该摄像头将立体视频实时传输至佩戴虚拟现实头显的操作员，使其可通过头部和身体动作控制相机姿态。系统提供沉浸式遥操作体验，支持双手第一人称操控，使操作员能动态探索场景并同步交互。我们在真实世界与仿真环境中对多种强调视角规划的任务进行了模仿学习实验，结果表明，由人类引导的主动视觉显著提升了模仿学习效果，在可见性受限任务中表现远超固定摄像头。
原文摘要 · Abstract (English)
Imitation learning has demonstrated significant potential in performing high-precision manipulation tasks using visual feedback. However, it is common practice in imitation learning for cameras to be fixed in place, resulting in issues like occlusion and limited field of view. Furthermore, cameras are often placed in broad, general locations, without an effective viewpoint specific to the robot's task. In this work, we investigate the utility of active vision (AV) for imitation learning and manipulation, in which, in addition to the manipulation policy, the robot learns an AV policy from human demonstrations to dynamically change the robot's camera viewpoint to obtain better information about its environment and the given task. We introduce AV-ALOHA, a new bimanual teleoperation robot system with AV, an extension of the ALOHA 2 robot system, incorporating an additional 7-DoF robot arm that only carries a stereo camera and is solely tasked with finding the best viewpoint. This camera streams stereo video to an operator wearing a virtual reality (VR) headset, allowing the operator to control the camera pose using head and body movements. The system provides an immersive teleoperation experience, with bimanual first-person control, enabling the operator to dynamically explore and search the scene and simultaneously interact with the environment. We conduct imitation learning experiments of our system both in real-world and in simulation, across a variety of tasks that emphasize viewpoint planning. Our results demonstrate the effectiveness of human-guided AV for imitation learning, showing significant improvements over fixed cameras in tasks with limited visibility. Project website: https://soltanilara.github.io/av-aloha/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。