构建首个大规模第一视角灵巧操作数据集，助力机器人学习人类手部动作。
EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Video
- 用Apple Vision Pro采集第一视角视频与3D手部关节数据
- 包含829小时视频、194种家务任务、精确手部追踪
- 适合研究机器人灵巧操作、视觉-运动模型的学者
灵巧操作的模仿学习面临数据稀缺问题。与自然语言和2D计算机视觉不同，目前缺乏大规模互联网级操作数据集。第一视角人类视频是一种潜在可扩展的数据源。然而，现有大规模数据集如Ego4D缺乏原生手部姿态标注且不聚焦物体操作。为此，我们使用Apple Vision Pro采集了EgoDex：迄今为止最大、最多样化的灵巧操作数据集。EgoDex包含829小时的第一视角视频，同步记录了每只手各关节的3D手部追踪数据，通过多相机校准与设备端SLAM实现高精度跟踪。数据涵盖194种不同桌面上的日常任务，从系鞋带到叠衣服，覆盖广泛的操作行为。此外，我们在该数据集上训练并系统评估了手部轨迹预测的模仿学习策略，引入新指标与基准以衡量该领域进展。通过发布此大规模数据集，我们希望推动机器人学、计算机视觉与基础模型的发展。EgoDex已公开下载：https://github.com/apple/ml-egodex。
原文摘要 · Abstract (English)
Imitation learning for manipulation has a well-known data scarcity problem. Unlike natural language and 2D computer vision, there is no Internet-scale corpus of data for dexterous manipulation. One appealing option is egocentric human video, a passively scalable data source. However, existing large-scale datasets such as Ego4D do not have native hand pose annotations and do not focus on object manipulation. To this end, we use Apple Vision Pro to collect EgoDex: the largest and most diverse dataset of dexterous human manipulation to date. EgoDex has 829 hours of egocentric video with paired 3D hand and finger tracking data collected at the time of recording, where multiple calibrated cameras and on-device SLAM can be used to precisely track the pose of every joint of each hand. The dataset covers a wide range of diverse manipulation behaviors with everyday household objects in 194 different tabletop tasks ranging from tying shoelaces to folding laundry. Furthermore, we train and systematically evaluate imitation learning policies for hand trajectory prediction on the dataset, introducing metrics and benchmarks for measuring progress in this increasingly important area. By releasing this large-scale dataset, we hope to push the frontier of robotics, computer vision, and foundation models. EgoDex is publicly available for download at https://github.com/apple/ml-egodex.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。