arXiv:2605.18553cs.CVcs.AI2026-05

基于观察质量动态调整双手动作估计,提升遮挡和离屏情况下的精度。

StableHand: Quality-Aware Flow Matching for World-Space Dual-Hand Motion Estimation from Egocentric Video

论文配图:StableHand: Quality-Aware Flow Matching for World-Space Dual-Hand Motion Estimation from Egocentric Video
图 1 · 摘自论文原文
  • 用四通道质量信号指导流匹配,区分可靠与不可靠观测
  • 在HOT3D和ARCTIC数据集上将误差降低20%-25%以上
  • 特别适合长时遮挡或视线丢失的日常动作分析场景

从第一人称视频中恢复双交互手在世界空间中的4D运动是监督机器人策略学习的基础能力,其中腕部轨迹追踪末端执行器,手指关节定义抓取姿态。该任务面临两大挑战:因头部运动导致手频繁长时间离开镜头视野,以及持续的手物交互造成严重遮挡。现有方法对噪声观测统一处理,未考虑每帧观测的质量可靠性,导致性能显著下降。本文核心洞察是:准确的世界空间手部运动估计与每帧观测质量紧密相关。为此,我们将来自现成手部姿态估计器的观测质量分解为四个通道:双手的腕部全局位移和手指关节。提出StableHand,一种基于质量感知的流匹配框架,通过学习的质量网络预测这四个通道的质量信号,并以四种方式融入流匹配过程:逐通道前向调度、质量调整的速度目标、基于质量的AdaLN调制的DiT去噪器,以及质量感知的ODE初始化。该统一生成流程保留高质量观测,同时利用学习到的双臂运动先验重建低质量观测。在包含长时缺失和持续遮挡的HOT3D与ARCTIC两个第一人称基准数据集上的实验表明,StableHand在所有指标上达到当前最优表现,相比最强基线将W-MPJPE降低20%-25%,在高度遮挡的ARCTIC序列上提升最显著。

原文摘要 · Abstract (English)

Recovering world space 4D motion of two interacting hands from egocentric video is a fundamental capability for supervising robot policy learning, where wrist trajectories track the end-effector and finger articulations specify the grasp pose. Two major challenges arise in this setting: hands frequently leave the camera view for extended periods due to head motion, and persistent hand-object interactions cause severe occlusions of one or both hands. Existing methods uniformly condition on noisy hand motion observations without accounting for their per-frame reliability, leading to substantial performance degradation. Our key insight is that accurate world space hand motion estimation is tightly coupled with the quality of per-frame hand observations. To this end, we decompose the quality of hand motion observations extracted from an off-the-shelf hand pose estimator into four channels: wrist global translation and finger articulations for both hands. We propose StableHand, a quality-aware flow-matching framework conditioned on these four-channel quality signals, which are predicted by a learned quality network. We naturally incorporate the quality signals into the flow-matching process through a per-channel forward schedule, a quality-adjusted velocity target, AdaLN modulation of the DiT denoiser, and a quality-aware ODE initialization. This unified generative process preserves high-quality observations while reconstructing unreliable ones using a learned bimanual motion prior. Experiments on HOT3D and ARCTIC, two egocentric benchmarks featuring long missing-hand spans and persistent hand-object occlusions, show that StableHand achieves state-of-the-art performance across all reported metrics, reducing W-MPJPE by 20-25% compared to the strongest baseline, with the largest gains on heavily occluded ARCTIC sequences.

动作估计双手建模质量感知流匹配

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。