无需标定即可融合激光雷达与摄像头,实时捕捉人体动作。
Sen-Cap: Sensor-Flexible and Noise-Resilient Human Motion Capture via LiDAR-Camera Integration

- 用统一空间估计人体姿态,不依赖传感器间标定。
- 在严重噪声下仍能稳定追踪动作,支持任意数量传感器。
- 适合运动分析、机器人、沉浸式环境等真实场景。
我们提出Sen-Cap,一种无需标定、抗噪声的3D人体动作捕捉框架,融合激光雷达与摄像头多模态数据。现有方法面临两大挑战:一是跨传感器对齐依赖显式标定,导致视角变化时误差传播,限制部署灵活性;二是面对严重噪声或部分传感器失效时性能下降。Sen-Cap引入统一的跨传感器姿态估计器,在人体中心空间重建局部姿态与形状,无需传感器间标定,支持任意数量传感器;同时设计抗噪声轨迹追踪器,通过迭代优化保持鲁棒性。该框架实现实时运行,在Human-M3和FreeMotion上达到领先指标,并在LiDARHuman26M和RELI11D上展现强跨域性能。其灵活性与鲁棒性为体育分析、野外机器人及大规模沉浸式环境中的动作捕捉开辟新可能。
原文摘要 · Abstract (English)
We propose Sen-Cap, a Sensor-Flexible and Noise-Resilient 3D human motion Capture framework that integrates multi-modal data from LiDAR and camera. While multi-modal sensors provide richer information than single-modal sensors, existing approaches still suffer from two core challenges. First, multi-modal alignment/matching across arbitrarily deployed sensors is typically handled by explicit calibration, which propagates errors under changing viewpoints and in turn constrains deployment to fixed, highly overlapped layouts. Second, prior methods degrade under severe noise or partial sensor failures, which are common in real-world environments. To address these challenges, Sen-Cap introduces a Unified Across-Sensor Motion Estimator that reconstructs local pose and shape in a human-centric space without calibrations between sensors, supporting a flexible number of sensors, as well as a Noise-Resistant Trajectory Tracker that maintains robustness under severe point cloud noise through iterative refinement. These sensor-flexible and noise-resilient features make Sen-Cap more practical in real-world deployment. Notably, operating in real time, Sen-Cap achieves state-of-the-art performance on major metrics on Human-M3 and FreeMotion, as well as strong cross-domain performance on LiDARHuman26M and RELI11D. This combination of flexibility and robustness opens new opportunities for motion capture in real-world scenarios, e.g. sports analytics, field robotics, and large-scale immersive environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。