用仿真先验提升雷达人体动作重建的泛化与数据效率
mmSimPrior: Learning Simulation Priors for Data-Efficient and Generalizable Real-World Radar-based Human Motion Reconstruction

- 分离信号、动作与雷达映射先验,通过物理感知随机化训练
- 仅需24段真实数据即比最强基线降低39%动作误差
- 适合缺乏标注数据的隐私敏感场景应用
毫米波雷达可实现无隐私泄露且光照鲁棒的人体动作重建,但训练通用模型通常需昂贵的配对雷达-动作数据。仿真可扩展监督,但物理模拟器无法完全复现真实世界的多径效应、杂波、硬件响应特性及距离相关的分辨率退化,造成仿真到现实的差距。我们提出mmSimPrior,一种将可迁移知识分解为信号、动作和雷达到动作映射先验的预训练框架。为学习可迁移的信号与动作先验,我们使用物理感知域随机化课程预训练多模态雷达编码器,以近似真实传播与采集级变化;联合时间标记器学习合理人体动作的离散先验。双模式映射模块可预测动作码分布以实现结构受限的零样本重建,或输出连续动作参数以适应少量真实数据。我们进一步构建了包含420万帧、3.1万序列的数据集套件,并引入无重叠设置,确保适应与测试集中无完全相同的主体-环境-位置-动作组合。在mmSimPrior-Real和RT-Pose上的实验显示一致提升:仅需24段配对真实序列,mmSimPrior-Reg在三个环境中相比最强基线降低24.7%-39.0%的MPJPE;mmSimPrior-Cls在无需微调情况下零样本MPJPE降低8.5%。
原文摘要 · Abstract (English)
Millimeter-wave (mmWave) radar enables privacy-preserving and illumination-robust human motion reconstruction, but training generalizable models typically requires costly paired radar-motion recordings. Simulation can scale such supervision, yet even physics-based simulators cannot fully reproduce real-world multipath, clutter, hardware-specific response statistics, or distance-dependent resolution degradation, leaving a sim-to-real gap. We present mmSimPrior, a simulation-pretrained framework that factorizes transferable knowledge into signal, motion, and radar-to-motion mapping priors. To learn transferable signal and motion priors, we pretrain a multimodal radar encoder with a physics-informed domain-randomization curriculum designed to mitigate the sim-to-real gap by approximating real-world propagation- and acquisition-level variations, while a joint-temporal tokenizer learns a discrete prior over plausible human motion. A dual-mode mapping module predicts either motion-code distributions for structurally constrained zero-shot reconstruction or continuous motion parameters for flexible adaptation from limited real data. We further construct a 4.2M-frame, 31K-sequence dataset suite and introduce a No-Overlap Setting that prevents any exact subject-environment-location-motion tuple from appearing in both the adaptation and test sets. Experiments on mmSimPrior-Real and RT-Pose demonstrate consistent gains: with only 24 paired real sequences, mmSimPrior-Reg reduces MPJPE by 24.7-39.0% over the strongest baseline across the three environments, while mmSimPrior-Cls reduces zero-shot MPJPE by 8.5% without fine-tuning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。