用手机+人体做数据采集,低成本构建通用场景的自视角视频库。
AoE: Always-on Egocentric Human Video Collection for Embodied AI
- 通过颈挂手机支架和云边协同架构,降低硬件门槛。
- 在多个下游任务中,高质量自视角数据显著提升模型泛化能力。
- 支持任何人随时随地采集,适合需要真实交互数据的研究者。
具身基础模型需要大规模、高质量的真实世界交互数据进行预训练与扩展。然而,现有数据采集方法存在基础设施成本高、硬件依赖复杂、交互范围有限等问题,难以规模化。事实上,人类本身就是理想的物理具身智能体。因此,通过全球分布的“人类代理”收集自视角真实交互数据,具有成本低、可持续的优势。为此,我们提出始终在线的自视角(AoE)数据采集系统,旨在通过利用人类自身及其智能手机,简化硬件依赖,实现低成本、高效、场景无关的真实世界交互数据采集,以缓解数据稀缺问题。具体而言,我们首先采用符合人体工学的颈挂式手机支架,通过云边协同架构实现低门槛的大规模自视角数据采集;其次,开发跨平台移动端应用,利用设备端计算进行实时处理,云端则运行自动化标注与筛选流水线,将原始视频转化为高质量训练数据;最后,AoE系统支持任何人、任何时间、任何地点的分布式自视角视频数据采集。我们在数据预处理质量及下游任务上评估了AoE,结果表明高质量自视角数据能显著提升模型在真实世界中的泛化性能。
原文摘要 · Abstract (English)
Embodied foundation models require large-scale, high-quality real-world interaction data for pre-training and scaling. However, existing data collection methods suffer from high infrastructure costs, complex hardware dependencies, and limited interaction scope, making scalable expansion challenging. In fact, humans themselves are ideal physically embodied agents. Therefore, obtaining egocentric real-world interaction data from globally distributed "human agents" offers advantages of low cost and sustainability. To this end, we propose the Always-on Egocentric (AoE) data collection system, which aims to simplify hardware dependencies by leveraging humans themselves and their smartphones, enabling low-cost, highly efficient, and scene-agnostic real-world interaction data collection to address the challenge of data scarcity. Specifically, we first employ an ergonomic neck-mounted smartphone holder to enable low-barrier, large-scale egocentric data collection through a cloud-edge collaborative architecture. Second, we develop a cross-platform mobile APP that leverages on-device compute for real-time processing, while the cloud hosts automated labeling and filtering pipelines that transform raw videos into high-quality training data. Finally, the AoE system supports distributed Ego video data collection by anyone, anytime, and anywhere. We evaluate AoE on data preprocessing quality and downstream tasks, demonstrating that high-quality egocentric data significantly boosts real-world generalization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。