从第一视角视频中重建手部世界坐标运动，解决相机动态下的高精度追踪问题。
HaWoR: World-Space Hand Motion Reconstruction from Egocentric Videos
- 分两步：先在相机坐标系重建手姿，再估计相机在世界坐标中的轨迹。
- 在多个第一视角数据集上实现手部和相机轨迹的领先性能。
- 提出新网络补全遮挡帧，提升复杂动作下手部轨迹连续性。
尽管3D手部姿态估计技术进步显著，现有方法仍主要聚焦于单图像在相机坐标系下的手部重建，忽略了手部在世界坐标系中的运动。这一局限性使其难以直接应用于第一视角视频场景，其中手部与相机持续运动。本文提出HaWoR，一种从第一视角视频中高保真重建手部世界空间运动的方法。通过将任务解耦：分别在相机坐标系中重建手部运动，并在世界坐标系中估计相机轨迹。为实现精确的相机轨迹估计，提出自适应第一视角SLAM框架，克服传统SLAM在剧烈相机运动下的不足，具备强鲁棒性。为确保手部运动轨迹稳定，即使手部移出视野范围，设计新型运动补全网络，有效恢复序列中缺失帧。通过大量定量与定性评估，证明HaWoR在不同第一视角基准数据集上，在手部运动重建与世界帧相机轨迹估计方面均达到当前最优性能。代码与模型已开源：https://hawor-project.github.io/。
原文摘要 · Abstract (English)
Despite the advent in 3D hand pose estimation, current methods predominantly focus on single-image 3D hand reconstruction in the camera frame, overlooking the world-space motion of the hands. Such limitation prohibits their direct use in egocentric video settings, where hands and camera are continuously in motion. In this work, we propose HaWoR, a high-fidelity method for hand motion reconstruction in world coordinates from egocentric videos. We propose to decouple the task by reconstructing the hand motion in the camera space and estimating the camera trajectory in the world coordinate system. To achieve precise camera trajectory estimation, we propose an adaptive egocentric SLAM framework that addresses the shortcomings of traditional SLAM methods, providing robust performance under challenging camera dynamics. To ensure robust hand motion trajectories, even when the hands move out of view frustum, we devise a novel motion infiller network that effectively completes the missing frames of the sequence. Through extensive quantitative and qualitative evaluations, we demonstrate that HaWoR achieves state-of-the-art performance on both hand motion reconstruction and world-frame camera trajectory estimation under different egocentric benchmark datasets. Code and models are available on https://hawor-project.github.io/ .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。