利用相似手形图像大规模预训练3D手部姿态估计,效果优于现有方法。
SiMHand: Mining Similar Hands for Large-Scale 3D Hand Pose Pre-training
- 通过对比学习挖掘非同一人但手姿相似的图像对进行预训练。
- 在FreiHand等数据集上提升15%~4%,超越当前最优方法PeCLR。
- 适合需要高精度手部姿态估计的研究者与开发者参考。
我们提出一种基于相似手形特征的大规模3D手部姿态预训练框架SiMHand。现有方法未充分利用来自真实场景视频的多样化手部图像。为此,我们从100DOH和Ego4D等人本中心视频中收集超过200万张手部图像,并设计基于对比学习的预训练方法。核心思想是聚焦于非同一人但手姿相似的图像对,在特征空间中拉近其表示。我们提出一种新型对比学习机制,根据样本间距离自适应加权损失,进一步提升性能。实验表明,该方法显著优于仅依赖单图增强生成正样本的传统对比学习,相比PeCLR在FreiHand上提升15%,DexYCB上10%,AssemblyHands上4%。代码已开源。
原文摘要 · Abstract (English)
We present a framework for pre-training of 3D hand pose estimation from in-the-wild hand images sharing with similar hand characteristics, dubbed SimHand. Pre-training with large-scale images achieves promising results in various tasks, but prior methods for 3D hand pose pre-training have not fully utilized the potential of diverse hand images accessible from in-the-wild videos. To facilitate scalable pre-training, we first prepare an extensive pool of hand images from in-the-wild videos and design our pre-training method with contrastive learning. Specifically, we collect over 2.0M hand images from recent human-centric videos, such as 100DOH and Ego4D. To extract discriminative information from these images, we focus on the similarity of hands: pairs of non-identical samples with similar hand poses. We then propose a novel contrastive learning method that embeds similar hand pairs closer in the feature space. Our method not only learns from similar samples but also adaptively weights the contrastive learning loss based on inter-sample distance, leading to additional performance gains. Our experiments demonstrate that our method outperforms conventional contrastive learning approaches that produce positive pairs sorely from a single image with data augmentation. We achieve significant improvements over the state-of-the-art method (PeCLR) in various datasets, with gains of 15% on FreiHand, 10% on DexYCB, and 4% on AssemblyHands. Our code is available at https://github.com/ut-vision/SiMHand.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。