用生成式掩码建模，让单图重建手部网格更准更鲁棒。
MaskHand: Generative Masked Modeling for Robust Hand Mesh Reconstruction in the Wild
- 通过掩码Transformer学习手部姿态的分布，生成多样且合理的3D网格。
- 在真实场景数据上，重建精度和鲁棒性均超越现有方法。
- 适合需要高精度手部建模的虚拟试衣、人机交互等应用。
从单张RGB图像重建3D手部网格面临复杂关节运动、自遮挡和深度模糊等挑战。传统判别方法因采用确定性映射，难以应对2D到3D映射的固有歧义。为此，我们提出MaskHand，一种用于手部网格恢复的新型生成式掩码模型，通过学习并采样模糊的2D到3D映射过程的概率分布，生成合理3D手部网格。MaskHand包含两个核心组件：(1) VQ-MANO，将3D手部动作编码为潜在空间中的离散姿态词元；(2) 上下文引导掩码Transformer，随机掩码部分姿态词元，学习其联合分布，条件依赖于受损词元序列、图像上下文及2D姿态线索。该分布支持推理时置信度引导采样，生成不确定性低、精度高的网格。在基准与真实世界数据集上的大量评估表明，MaskHand在3D手部网格重建中实现了最先进的准确率、鲁棒性和真实性。
原文摘要 · Abstract (English)
Reconstructing a 3D hand mesh from a single RGB image is challenging due to complex articulations, self-occlusions, and depth ambiguities. Traditional discriminative methods, which learn a deterministic mapping from a 2D image to a single 3D mesh, often struggle with the inherent ambiguities in 2D-to-3D mapping. To address this challenge, we propose MaskHand, a novel generative masked model for hand mesh recovery that synthesizes plausible 3D hand meshes by learning and sampling from the probabilistic distribution of the ambiguous 2D-to-3D mapping process. MaskHand consists of two key components: (1) a VQ-MANO, which encodes 3D hand articulations as discrete pose tokens in a latent space, and (2) a Context-Guided Masked Transformer that randomly masks out pose tokens and learns their joint distribution, conditioned on corrupted token sequence, image context, and 2D pose cues. This learned distribution facilitates confidence-guided sampling during inference, producing mesh reconstructions with low uncertainty and high precision. Extensive evaluations on benchmark and real-world datasets demonstrate that MaskHand achieves state-of-the-art accuracy, robustness, and realism in 3D hand mesh reconstruction. Project website: https://m-usamasaleem.github.io/publication/MaskHand/MaskHand.html.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。