用混合模型提升视觉惯性人体姿态估计，实时且更准。
VIMCAN: Visual-Inertial 3D Human Pose Estimation with Hybrid Mamba-Cross-Attention Network
- 结合Mamba与交叉注意力，兼顾时序效率与空间关系建模
- 在TotalCapture上达17.2毫米平均关节误差，3DPW上45.3毫米
- 支持消费级硬件实时推理，适合移动/穿戴设备应用
深度学习的进展显著提升了多模态3D人体姿态估计（HPE）的精度。然而，当前最先进（SOTA）的HPE流程仍依赖Transformer，其二次复杂度导致长序列实时处理不现实。Mamba通过选择性状态空间建模解决了这一问题，实现高效序列处理而不损失表达能力。但其在多模态场景中难以捕捉复杂空间依赖。为此，我们提出VIMCAN，一种融合Mamba与交叉注意力的混合架构，实现鲁棒的视觉-惯性融合及人体姿态估计，输入为RGB关键点与可穿戴IMU数据。通过Mamba动态参数化进行时序建模，交叉注意力提取空间依赖，VIMCAN在TotalCapture上取得17.2毫米的均方关节位置误差（MPJPE），在3DPW上为45.3毫米。该方法优于以往基于Transformer及其他SOTA方案，同时可在消费级硬件上实现超过60帧每秒的实时推理。源代码已公开于GitHub。
原文摘要 · Abstract (English)
The rapid advances in deep learning have significantly enhanced the accuracy of multimodal 3D human pose estimation (HPE). However, the state-of-the-art (SOTA) HPE pipelines still rely on Transformers, whose quadratic complexity makes real-time processing for long sequences impractical. Mamba addresses this issue through selective state-space modeling, enabling efficient sequence processing without sacrificing representational power. Nevertheless, it struggles to capture complex spatial dependencies in multimodal settings. To bridge this gap, we propose VIMCAN, a hybrid architecture that combines the efficient sequence modeling of Mamba with the spatial reasoning of Cross-Attention, and performs robust visual-inertial fusion and human pose estimation between RGB keypoints and wearable IMU data. By leveraging Mamba's dynamic parameterization for temporal modeling and Attention for spatial dependency extraction, VIMCAN achieves superior accuracy, with mean per-joint position errors (MPJPE) of 17.2 mm on TotalCapture and 45.3 mm on 3DPW. VIMCAN outperforms prior Transformer-based and other SOTA approaches while supporting real-time inference at over 60 frames per second on consumer-grade hardware. The source code is available at \href{https://github.com/Eddieyzp/VIMCAN}{this GitHub repository}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。