提出PoseMoE模型,用专家混合机制分离2D姿态与深度特征,提升单目3D人体姿态估计精度。
PoseMoE: Mixture-of-Experts Network for Monocular 3D Human Pose Estimation
- 采用专家混合架构,分别处理2D姿态和深度特征,避免不确定性干扰
- 在Human3.6M、MPI-INF-3DHP、3DPW上均超越传统方法,最高提升5.2% PCK
- 适合需要高精度3D姿态估计的动捕、虚拟现实场景
基于提升的方法在单目3D人体姿态估计中占据主导地位,利用检测到的2D姿态作为中间表示。其中,3D姿态的2D分量受益于检测结果,而深度分量则需从零开始估计。现有方法将检测到的2D姿态与未知深度编码在同一特征空间,显式引入深度不确定性,限制了整体估计精度。本文揭示深度表示在估计过程中的关键作用:当深度初始状态完全未知时,联合编码2D姿态与深度特征会损害估计效果;反之,若通过网络预估使深度进入更可靠状态,则联合编码有益。为此,我们提出面向单目3D姿态估计的专家混合网络PoseMoE。其核心包括:(1) 专家混合网络,由专门模块分别精炼已检测的2D姿态特征并学习深度特征,解耦2D与深度特征编码过程,降低不确定深度对2D特征的影响;(2) 跨专家知识聚合模块,通过双向映射在2D姿态与深度间传递时空上下文信息。大量实验表明,PoseMoE在三个主流数据集Human3.6M、MPI-INF-3DHP和3DPW上均优于传统提升方法。
原文摘要 · Abstract (English)
The lifting-based methods have dominated monocular 3D human pose estimation by leveraging detected 2D poses as intermediate representations. The 2D component of the final 3D human pose benefits from the detected 2D poses, whereas its depth counterpart must be estimated from scratch. The lifting-based methods encode the detected 2D pose and unknown depth in an entangled feature space, explicitly introducing depth uncertainty to the detected 2D pose, thereby limiting overall estimation accuracy. This work reveals that the depth representation is pivotal for the estimation process. Specifically, when depth is in an initial, completely unknown state, jointly encoding depth features with 2D pose features is detrimental to the estimation process. In contrast, when depth is initially refined to a more dependable state via network-based estimation, encoding it together with 2D pose information is beneficial. To address this limitation, we present a Mixture-of-Experts network for monocular 3D pose estimation named PoseMoE. Our approach introduces: (1) A mixture-of-experts network where specialized expert modules refine the well-detected 2D pose features and learn the depth features. This mixture-of-experts design disentangles the feature encoding process for 2D pose and depth, therefore reducing the explicit influence of uncertain depth features on 2D pose features. (2) A cross-expert knowledge aggregation module is proposed to aggregate cross-expert spatio-temporal contextual information. This step enhances features through bidirectional mapping between 2D pose and depth. Extensive experiments show that our proposed PoseMoE outperforms the conventional lifting-based methods on three widely used datasets: Human3.6M, MPI-INF-3DHP, and 3DPW.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。