用轻量Transformer实现单目3D人体姿态实时估计,支持有无相机标定场景。
PriorFormer: A Transformer for Real-time Monocular 3D Human Pose Estimation with Versatile Geometric Priors
- 引入肢体长度和相机内参等几何先验,通过掩码机制适应缺失信息
- 平均3D关节定位误差仅36mm,比当前最优结果提升0.5厘米,计算耗时低于400μs
- 适合嵌入式设备部署,兼顾精度与实时性,适用于真实环境应用
本文提出一种基于轻量级Transformer的3D人体姿态估计算法(Lifter),通过单目摄像头将短时序2D关节点位置映射为3D姿态。模型输入包含肢体段长和相机内参等几何先验,可在已标定与未标定环境下统一运行。通过掩码机制,模型在训练和推理中可忽略缺失的先验信息,实现单一通用网络适配多种部署场景。模型使用AMASS数据集的3D关键点及随机采样相机位姿与内参生成的合成2D数据进行训练,并与仅在完整先验下训练的专家模型对比。消融实验表明,相机与段长先验均能提升性能;即使所有先验可用,该通用模型仍优于专家模型,且在先验缺失时保持高精度。平均3D关节中心定位误差低至36mm,较现有方法提升0.5厘米,同时在GPU上仅需380μs,在CPU上1800μs,具备嵌入式部署能力。
原文摘要 · Abstract (English)
This paper proposes a new lightweight Transformer-based lifter that maps short sequences of human 2D joint positions to 3D poses using a single camera. The proposed model takes as input geometric priors including segment lengths and camera intrinsics and is designed to operate in both calibrated and uncalibrated settings. To this end, a masking mechanism enables the model to ignore missing priors during training and inference. This yields a single versatile network that can adapt to different deployment scenarios, from fully calibrated lab environments to in-the-wild monocular videos without calibration. The model was trained using 3D keypoints from AMASS dataset with corresponding 2D synthetic data generated by sampling random camera poses and intrinsics. It was then compared to an expert model trained, only on complete priors, and the validation was done by conducting an ablation study. Results show that both, camera and segment length priors, improve performance and that the versatile model outperforms the expert, even when all priors are available, and maintains high accuracy when priors are missing. Overall the average 3D joint center positions estimation accuracy was as low as 36mm improving state of the art by half a centimeter and at a much lower computational cost. Indeed, the proposed model runs in 380$μ$s on GPU and 1800$μ$s on CPU, making it suitable for deployment on embedded platforms and low-power devices.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。