让机器人像人一样看再动，通过视觉引导动作完成任务。
Eye, Robot: Learning to Look to Act with a BC-RL Perception-Action Loop
- 用强化学习训练眼球注视策略，与手部动作联合优化。
- 在5个全景操作任务中实现有效手眼协调，提升大范围操作能力。
- 模拟人眼聚焦机制，低算力下稳定追踪目标、忽略干扰物。
人类不会被动观察世界，而是主动注视以指导行动。受此启发，我们提出EyeRobot——一种具有自主注视行为的机器人系统，其眼动行为由完成真实任务的需求自然涌现。我们设计了一个可自由旋转的机械眼球，并通过配对360°摄像头的遥操作数据，在支持任意眼球视角渲染的仿真环境中生成带眼动轨迹的机器人演示。随后引入行为克隆-强化学习（BC-RL）循环：手部代理（BC）基于渲染的眼球视角进行训练，眼球代理（RL）则在手部预测正确动作时获得奖励。在此机制下，手眼协调自发形成，眼球会注视有助于任务完成的区域。系统采用类中央凹的策略架构，在小计算开销下实现高分辨率感知，同时促使更稳定的凝视、更强的目标追踪能力及对干扰物的抑制。我们在五个环绕机器人手臂的全景工作区操作任务上评估了EyeRobot，实验表明其能有效利用单个摄像头完成大范围操作任务中的手眼协同。项目视频见：https://www.eyerobot.net/
原文摘要 · Abstract (English)
Humans do not passively observe the visual world -- we actively look in order to act. Motivated by this principle, we introduce EyeRobot, a robotic system with gaze behavior that emerges from the need to complete real-world tasks. We develop a mechanical eyeball that can freely rotate to observe its surroundings and train a gaze policy to control it using reinforcement learning. We accomplish this by first collecting teleoperated demonstrations paired with a 360 camera. This data is imported into a simulation environment that supports rendering arbitrary eyeball viewpoints, allowing episode rollouts of eye gaze on top of robot demonstrations. We then introduce a BC-RL loop to train the hand and eye jointly: the hand (BC) agent is trained from rendered eye observations, and the eye (RL) agent is rewarded when the hand produces correct action predictions. In this way, hand-eye coordination emerges as the eye looks towards regions which allow the hand to complete the task. EyeRobot implements a foveal-inspired policy architecture allowing high resolution with a small compute budget, which we find also leads to the emergence of more stable fixation as well as improved ability to track objects and ignore distractors. We evaluate EyeRobot on five panoramic workspace manipulation tasks requiring manipulation in an arc surrounding the robot arm. Our experiments suggest EyeRobot exhibits hand-eye coordination behaviors which effectively facilitate manipulation over large workspaces with a single camera. See project site for videos: https://www.eyerobot.net/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。