一阶段完成手部检测与3D重建,精度领先且无累计误差。
HandOS: 3D Hand Reconstruction in One Stage
- 用冻结检测器为基础,融合2D/3D关键点估计模块。
- 在FreiHand上达5.0 PA-MPJPE,HInt-Ego4D上64.6% [email protected]。
- 无需左右手分类,适合实时交互应用。
现有手部重建方法多采用多阶段框架,包括检测、左右手分类和姿态估计,导致计算冗余与误差累积。本文提出HandOS,一种端到端的一阶段3D手部重建框架。核心思路是利用冻结的检测器作为基础,引入辅助模块实现2D与3D关键点估计,将姿态估计能力融入检测框架,同时避免以左右手类别为前提。具体设计交互式2D-3D解码器:2D关节语义由检测线索获得,3D表示则从2D关节提升而来;并引入层级注意力机制,同步建模2D关节、3D顶点与相机位移。最终实现手部检测、2D姿态估计与3D网格重建的一阶段集成,克服多阶段缺陷。在公开基准上达到最优性能,FreiHand上5.0 PA-MPJPE,HInt-Ego4D上64.6% [email protected]。项目页面:idea-research.github.io/HandOSweb。
原文摘要 · Abstract (English)
Existing approaches of hand reconstruction predominantly adhere to a multi-stage framework, encompassing detection, left-right classification, and pose estimation. This paradigm induces redundant computation and cumulative errors. In this work, we propose HandOS, an end-to-end framework for 3D hand reconstruction. Our central motivation lies in leveraging a frozen detector as the foundation while incorporating auxiliary modules for 2D and 3D keypoint estimation. In this manner, we integrate the pose estimation capacity into the detection framework, while at the same time obviating the necessity of using the left-right category as a prerequisite. Specifically, we propose an interactive 2D-3D decoder, where 2D joint semantics is derived from detection cues while 3D representation is lifted from those of 2D joints. Furthermore, hierarchical attention is designed to enable the concurrent modeling of 2D joints, 3D vertices, and camera translation. Consequently, we achieve an end-to-end integration of hand detection, 2D pose estimation, and 3D mesh reconstruction within a one-stage framework, so that the above multi-stage drawbacks are overcome. Meanwhile, the HandOS reaches state-of-the-art performances on public benchmarks, e.g., 5.0 PA-MPJPE on FreiHand and 64.6\% [email protected] on HInt-Ego4D. Project page: idea-research.github.io/HandOSweb.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。