让手姿估计模型在新场景下更准,不依赖标注数据自适应。
Towards Egocentric 3D Hand Pose Estimation in Unseen Domains
- 在虚拟相机空间估算深度,消除镜头参数影响。
- 测试时用3D一致性损失优化,误差比之前降低71%和41%。
- 只需少量数据就能超越主流方法,适合实际部署场景。
我们提出V-HPOT,一种提升从第一人称图像中进行跨域3D手部姿态估计性能的新方法。现有先进方法在相同域内表现优异,但在新环境泛化能力差,主要受限于训练数据不足与深度感知缺陷——对特定相机内参过拟合。本方法通过在由焦距和图像尺寸归一化的虚拟相机空间中估计关键点的z坐标,实现与相机无关的深度预测。进一步利用此不变性,提出一种自监督的测试时优化策略,在推理阶段通过施加预测姿态与空间缩放后姿态间的3D一致性损失,使模型无需真实标注即可适应目标域特性。实验表明,V-HPOT在跨域场景中显著提升性能:在H2O数据集上平均姿态误差降低71%,在AssemblyHands数据集上降低41%。相比现有最优方法,其在所有数据集上均优于所有单阶段方法,并与双阶段方法竞争,且仅需约1/3.5至1/14的数据量。
原文摘要 · Abstract (English)
We present V-HPOT, a novel approach for improving the cross-domain performance of 3D hand pose estimation from egocentric images across diverse, unseen domains. State-of-the-art methods demonstrate strong performance when trained and tested within the same domain. However, they struggle to generalise to new environments due to limited training data and depth perception -- overfitting to specific camera intrinsics. Our method addresses this by estimating keypoint z-coordinates in a virtual camera space, normalised by focal length and image size, enabling camera-agnostic depth prediction. We further leverage this invariance to camera intrinsics to propose a self-supervised test-time optimisation strategy that refines the model's depth perception during inference. This is achieved by applying a 3D consistency loss between predicted and in-space scale-transformed hand poses, allowing the model to adapt to target domain characteristics without requiring ground truth annotations. V-HPOT significantly improves 3D hand pose estimation performance in cross-domain scenarios, achieving a 71% reduction in mean pose error on the H2O dataset and a 41% reduction on the AssemblyHands dataset. Compared to state-of-the-art methods, V-HPOT outperforms all single-stage approaches across all datasets and competes closely with two-stage methods, despite needing approximately x3.5 to x14 less data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。