用语言对齐动作信息,提升复杂场景下3D人体姿态估计精度
LangPose: Language-Aligned Motion for Robust 3D Human Pose Estimation
- 通过文本与动作嵌入对齐,利用语义信息解决深度模糊问题
- 在Human3.6M上达36.7mm MPJPE(输入为检测2D姿态)
- 适合需要高鲁棒性3D姿态估计的运动分析、虚拟现实应用
2D到3D人体姿态重建因深度模糊和遮挡而成为病态问题。仅依赖空间和时间一致性的方法在严重遮挡或高动态动作下表现不足。本文提出LangPose框架,通过将动作嵌入与细粒度动作标签的文本嵌入对齐,引入语义信息辅助判断。该框架分两阶段:预训练阶段同时学习动作识别与从带噪、掩码2D姿态中重建3D姿态;微调阶段在无动作标签的真实3D数据集上进一步优化。同时,模型引入身体部位掩码与时间窗口掩码,促使模型在空间-时间一致性不可靠时依赖语义线索。实验表明,LangPose在公开数据集上达到当前最优性能,于Human3.6M上使用检测2D姿态输入时取得36.7mm MPJPE,于MPI-INF-3DHP上使用真值2D姿态输入时达15.5mm MPJPE。
原文摘要 · Abstract (English)
2D-to-3D human pose lifting is an ill-posed problem due to depth ambiguity and occlusion. Existing methods relying on spatial and temporal consistency alone are insufficient to resolve these problems especially in the presence of significant occlusions or high dynamic actions. Semantic information, however, offers a complementary signal that can help disambiguate such cases. To this end, we propose LangPose, a framework that leverages action knowledge by aligning motion embeddings with text embeddings of fine-grained action labels. LangPose operates in two stages: pretraining and fine-tuning. In the pretraining stage, the model simultaneously learns to recognize actions and reconstruct 3D poses from masked and noisy 2D poses. During the fine-tuning stage, the model is further refined using real-world 3D human pose estimation datasets without action labels. Additionally, our framework incorporates masked body parts and masked time windows in motion modeling, encouraging the model to leverage semantic information when spatial and temporal consistency is unreliable. Experiments demonstrate the effectiveness of LangPose, achieving SOTA level performance in 3D pose estimation on public datasets, including Human3.6M and MPI-INF-3DHP. Specifically, LangPose achieves an MPJPE of 36.7mm on Human3.6M with detected 2D poses as input and 15.5mm on MPI-INF-3DHP with ground-truth 2D poses as input.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。