用双雷达融合技术提升毫米波雷达人体姿态估计精度
milliMamba: Specular-Aware Human Pose Estimation via Dual mmWave Radar with Multi-Frame Mamba Fusion
- 双雷达输入+跨视角融合Mamba编码器，高效提取时序特征
- 在TransHuPR和HuPR数据集上分别提升11.0和14.6的AP指标
- 适合隐私敏感或光照复杂场景下的实时姿态识别应用
毫米波雷达为人体姿态估计（HPE）提供了隐私保护且不受光照影响的替代方案。然而，由于镜面反射导致信号稀疏，从雷达信号中提取鲁棒特征极具挑战。为此，我们提出milliMamba，一种基于雷达的2D人体姿态估计框架，联合建模特征提取与解码阶段的时空依赖关系。针对雷达输入高维特性，采用跨视图融合Mamba编码器，以线性复杂度高效提取长序列时空特征；随后通过时空交叉注意力解码器预测多帧关节坐标。该时空建模流程可利用邻近帧与关节的上下文信息，推断因镜面反射缺失的关节点。为增强运动平滑性，训练时引入速度损失与标准关键点损失。在TransHuPR和HuPR数据集上的实验表明，本方法分别较基线提升11.0 AP和14.6 AP，同时保持合理计算复杂度。
原文摘要 · Abstract (English)
Millimeter-wave radar offers a privacy-preserving and lighting-invariant alternative to RGB sensors for Human Pose Estimation (HPE) task. However, the radar signals are often sparse due to specular reflection, making the extraction of robust features from radar signals highly challenging. To address this, we present milliMamba, a radar-based 2D human pose estimation framework that jointly models spatio-temporal dependencies across both the feature extraction and decoding stages. Specifically, given the high dimensionality of radar inputs, we adopt a Cross-View Fusion Mamba encoder to efficiently extract spatio-temporal features from longer sequences with linear complexity. A Spatio-Temporal-Cross Attention decoder then predicts joint coordinates across multiple frames. Together, this spatio-temporal modeling pipeline enables the model to leverage contextual cues from neighboring frames and joints to infer missing joints caused by specular reflections. To reinforce motion smoothness, we incorporate a velocity loss alongside the standard keypoint loss during training. Experiments on the TransHuPR and HuPR datasets demonstrate that our method achieves significant performance improvements, exceeding the baselines by 11.0 AP and 14.6 AP, respectively, while maintaining reasonable complexity. Code: https://github.com/NYCU-MAPL/milliMamba
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。