用稀疏传感器数据实时生成逼真人体动作，提升虚拟现实体验
Mem-MLP: Real-Time 3D Human Motion Generation from Sparse Inputs
- 基于MLP架构，引入记忆模块动态补全缺失动作数据
- 在移动头显上实现72帧/秒，误差显著低于现有方法
- 适合实时动作捕捉与轻量化设备部署
真实流畅的全身动作追踪对沉浸式增强现实/虚拟现实应用至关重要。现有系统主要依赖头戴设备和控制器追踪头部与手部，导致全身重建不完整。一种可行方案是通过神经网络模型，从有限传感器采集的稀疏输入生成完整身体动作。本文提出一种基于多层感知机（MLP）的新型方法，其结构包含残差连接和一种新设计的内存块（Memory-Block）。Memory-Block使用可训练的代码向量表示缺失传感器数据，并结合前一时刻的稀疏信号，提升动作的时间一致性。此外，将该方法建模为多任务学习问题，使MLP主干学习更鲁棒的表征以提升精度。实验表明，该方法显著降低预测误差，且在移动头显上达到72 FPS，有效优化了准确率与运行速度的权衡。
原文摘要 · Abstract (English)
Realistic and smooth full-body tracking is crucial for immersive AR/VR applications. Existing systems primarily track head and hands via Head Mounted Devices (HMDs) and controllers, making the 3D full-body reconstruction in-complete. One potential approach is to generate the full-body motions from sparse inputs collected from limited sensors using a Neural Network (NN) model. In this paper, we propose a novel method based on a multi-layer perceptron (MLP) backbone that is enhanced with residual connections and a novel NN-component called Memory-Block. In particular, Memory-Block represents missing sensor data with trainable code-vectors, which are combined with the sparse signals from previous time instances to improve the temporal consistency. Furthermore, we formulate our solution as a multi-task learning problem, allowing our MLP-backbone to learn robust representations that boost accuracy. Our experiments show that our method outperforms state-of-the-art baselines by substantially reducing prediction errors. Moreover, it achieves 72 FPS on mobile HMDs that ultimately improves the accuracy-running time tradeoff.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。