用手机手表等设备实现任意组合的全身动作捕捉。
Towards Real-World Wearable Motion Reconstruction
- 基于多种可穿戴设备构建统一动作重建框架。
- 在50种活动中实现高精度动作还原，支持缺损传感器补全。
- 揭示不同设备间互补性，指导实际部署选型。
可穿戴设备的普及带来了一个全新的动作捕捉挑战：如何从任意组合的传感器中重建全身运动。现有研究多依赖固定配置（如惯性测量单元套装或头显中心系统），难以跨配置泛化。本文主张优先使用轻便无感的消费级设备，如智能手机、智能手表、智能眼镜和智能鞋垫，并研究它们之间的协同关系。为此，我们做出三项贡献：第一，构建一个大规模多模态数据集，同步消费者级传感器与真实3D动作数据，覆盖50种多样化活动，包括日常任务、体育运动和社交互动；第二，提出WHIP基线生成模型，可从任意可用传感器子集重建动作，对缺失模态具有鲁棒性并生成物理合理的运动轨迹；第三，系统研究传感器互补性，量化不同模态间的协同增益。代码与数据集已公开于https://vcai.mpi-inf.mpg.de/projects/WHIP/
原文摘要 · Abstract (English)
The modern-day surge in popularity of wearable devices poses a fundamentally unique motion capture problem: reconstructing full-body movement from any set of sensing hardware worn at a given moment. Yet, most research efforts assume fixed sensor configurations (e.g. IMU suits or HMD-centric rigs) and cannot generalize across them. In contrast, we argue that motion capture should prioritize unobtrusive and lightweight devices such as smartphones, smartwatches, smart glasses, and smart insoles, and study the interplay between them. To this end, we make three contributions. First, we present a large-scale multi-modal dataset synchronizing these consumer-grade sensors with ground-truth 3D motion, spanning 50 diverse activities including everyday tasks, sports, and social interactions. Second, we propose WHIP, a baseline generative model that reconstructs motion from arbitrary subsets of available sensors, robustly handling missing modalities and producing physically plausible motions. Third, we conduct a systematic study of sensor complementarity, quantifying how different modalities complement one another. Code and dataset are available at https://vcai.mpi-inf.mpg.de/projects/WHIP/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。