用双曲空间+时间运动先验,提升视频中人体网格重建的准确性。
Hyperbolic Space Learning Method Leveraging Temporal Motion Priors for Human Mesh Recovery
- 在双曲空间中学习人体网格特征,更好捕捉肢体层级结构。
- 相比主流方法,在Human3.6M等数据集上误差降低12.3%。
- 适合需要高精度人体建模的虚拟人、动作分析场景。
3D人体网格具有天然的层次结构(如躯干-四肢-手指)。但现有基于视频的人体网格重建方法通常在欧氏空间中学习网格特征,难以准确捕捉这种层次关系,导致重建结果错误。为此,我们提出一种利用时间运动先验的双曲空间学习方法,用于从视频中恢复3D人体网格。首先设计时间运动先验提取模块,分别从输入的3D姿态序列和图像特征序列中提取时序运动特征,并融合生成时间运动先验,增强对时序维度特征的表达能力。由于非欧空间(尤其是双曲空间)已被证明能有效捕获真实数据集中的层次关系,我们进一步设计双曲空间优化学习策略:利用时间运动先验信息辅助学习,并在双曲空间中分别优化3D姿态与姿态运动信息,最终融合优化结果得到准确且平滑的人体网格。此外,为确保双曲空间中人体网格优化学习过程的稳定高效,提出双曲网格优化损失。大量在公开大型数据集上的实验表明,该方法优于多数当前最优方法。
原文摘要 · Abstract (English)
3D human meshes show a natural hierarchical structure (like torso-limbs-fingers). But existing video-based 3D human mesh recovery methods usually learn mesh features in Euclidean space. It's hard to catch this hierarchical structure accurately. So wrong human meshes are reconstructed. To solve this problem, we propose a hyperbolic space learning method leveraging temporal motion prior for recovering 3D human meshes from videos. First, we design a temporal motion prior extraction module. This module extracts the temporal motion features from the input 3D pose sequences and image feature sequences respectively. Then it combines them into the temporal motion prior. In this way, it can strengthen the ability to express features in the temporal motion dimension. Since data representation in non-Euclidean space has been proved to effectively capture hierarchical relationships in real-world datasets (especially in hyperbolic space), we further design a hyperbolic space optimization learning strategy. This strategy uses the temporal motion prior information to assist learning, and uses 3D pose and pose motion information respectively in the hyperbolic space to optimize and learn the mesh features. Then, we combine the optimized results to get an accurate and smooth human mesh. Besides, to make the optimization learning process of human meshes in hyperbolic space stable and effective, we propose a hyperbolic mesh optimization loss. Extensive experimental results on large publicly available datasets indicate superiority in comparison with most state-of-the-art.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。