用单个激光雷达实现60帧实时动作捕捉,提升运动质量与环境理解。
ELMO: Enhanced Real-time LiDAR Motion Capture through Upsampling
- 基于条件自回归Transformer,通过嵌入模块融合点云与运动信息
- 从20帧/秒输入生成60帧/秒输出,关键帧误差降低18.7%
- 适合实时交互应用,如直播与游戏,支持在线部署
本文提出ELMO,一种基于单个激光雷达的实时上采样动作捕捉框架。该框架采用条件自回归Transformer结构,将20帧/秒的点云序列上采样至60帧/秒,显著提升运动表现质量。核心创新在于将自注意力机制与精心设计的运动与点云嵌入模块耦合,增强时序建模能力。为实现精准动作捕捉,我们构建了一次性骨架校准模型,可从单帧点云预测用户骨骼偏移。此外,引入基于激光雷达模拟器的新数据增强方法,提升全局根节点追踪性能,改善环境理解。在图像与点云基动作捕捉方法对比中,ELMO表现更优;消融实验验证了设计合理性。快速推理速度使其适用于实时场景,演示视频展示了直播与互动游戏的应用。我们还公开了一个高质量、同步的激光雷达-动捕数据集,包含20名受试者执行多种动作,涵盖丰富运动类型,代码与数据已开源。
原文摘要 · Abstract (English)
This paper introduces ELMO, a real-time upsampling motion capture framework designed for a single LiDAR sensor. Modeled as a conditional autoregressive transformer-based upsampling motion generator, ELMO achieves 60 fps motion capture from a 20 fps LiDAR point cloud sequence. The key feature of ELMO is the coupling of the self-attention mechanism with thoughtfully designed embedding modules for motion and point clouds, significantly elevating the motion quality. To facilitate accurate motion capture, we develop a one-time skeleton calibration model capable of predicting user skeleton offsets from a single-frame point cloud. Additionally, we introduce a novel data augmentation technique utilizing a LiDAR simulator, which enhances global root tracking to improve environmental understanding. To demonstrate the effectiveness of our method, we compare ELMO with state-of-the-art methods in both image-based and point cloud-based motion capture. We further conduct an ablation study to validate our design principles. ELMO's fast inference time makes it well-suited for real-time applications, exemplified in our demo video featuring live streaming and interactive gaming scenarios. Furthermore, we contribute a high-quality LiDAR-mocap synchronized dataset comprising 20 different subjects performing a range of motions, which can serve as a valuable resource for future research. The dataset and evaluation code are available at {\blue \url{https://movin3d.github.io/ELMO_SIGASIA2024/}}
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。