用生成模型解决单目人体3D重建的模糊性问题,效果超越现有方法。
GenHMR: Generative Human Mesh Recovery
- 将单图人体重建转为图像条件生成任务,用离散令牌建模姿态分布
- 通过掩码Transformer学习姿态令牌的概率分布,推理时采样生成多解
- 结合2D姿态引导优化,提升重建精度,适合动作捕捉与虚拟人应用
人体网格恢复(HMR)在健康、艺术和娱乐等领域至关重要。单目图像的HMR因深度模糊和遮挡问题属于病态问题。传统确定性方法仅输出单一预测,而概率方法虽尝试生成多个合理3D重构,但性能常落后于确定性方法。本文提出GenHMR,一种新型生成框架,将单目HMR重定义为图像条件生成任务,显式建模并缓解2D到3D映射中的不确定性。GenHMR包含两个核心组件:(1) 姿态分词器,将3D人体姿态转换为潜在空间中的离散令牌序列;(2) 图像条件掩码Transformer,基于输入图像提示和随机掩码的令牌序列,学习姿态令牌的概率分布。推理时,模型从学习的条件分布中采样,迭代解码高置信度姿态令牌,降低3D重建不确定性。为进一步优化,提出2D姿态引导的精修技术,在潜在空间直接微调解码后的姿态令牌,使投影的3D身体网格与2D姿态线索对齐。在基准数据集上的实验表明,GenHMR显著优于当前最优方法。
原文摘要 · Abstract (English)
Human mesh recovery (HMR) is crucial in many computer vision applications; from health to arts and entertainment. HMR from monocular images has predominantly been addressed by deterministic methods that output a single prediction for a given 2D image. However, HMR from a single image is an ill-posed problem due to depth ambiguity and occlusions. Probabilistic methods have attempted to address this by generating and fusing multiple plausible 3D reconstructions, but their performance has often lagged behind deterministic approaches. In this paper, we introduce GenHMR, a novel generative framework that reformulates monocular HMR as an image-conditioned generative task, explicitly modeling and mitigating uncertainties in the 2D-to-3D mapping process. GenHMR comprises two key components: (1) a pose tokenizer to convert 3D human poses into a sequence of discrete tokens in a latent space, and (2) an image-conditional masked transformer to learn the probabilistic distributions of the pose tokens, conditioned on the input image prompt along with randomly masked token sequence. During inference, the model samples from the learned conditional distribution to iteratively decode high-confidence pose tokens, thereby reducing 3D reconstruction uncertainties. To further refine the reconstruction, a 2D pose-guided refinement technique is proposed to directly fine-tune the decoded pose tokens in the latent space, which forces the projected 3D body mesh to align with the 2D pose clues. Experiments on benchmark datasets demonstrate that GenHMR significantly outperforms state-of-the-art methods. Project website can be found at https://m-usamasaleem.github.io/publication/GenHMR/GenHMR.html
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。