arXiv:2607.11221cs.CVcs.AI2026-07

用流匹配生成4D手部动作,实时稳定恢复单目视频中的手姿。

HandFlow: Fully Generative 4D Hand Recovery with Flow Matching

论文配图:HandFlow: Fully Generative 4D Hand Recovery with Flow Matching
图 1 · 摘自论文原文
  • 基于流匹配的生成式框架,一次性去噪整段视频的手部参数。
  • 世界空间误差降30%以上,加速度误差最低,时间连续性极佳。
  • 适合需要高精度、高帧率手部重建的交互与动画应用。

单目4D手部重建仍具挑战:逐帧判别回归器缺乏时序上下文,易产生抖动;传统时序模型多为确定性回归,难以应对遮挡和运动模糊带来的歧义。生成建模通过学习合理的手部运动先验,在视觉证据缺失或不可靠时仍能实现连贯恢复。为此,我们提出HandFlow,一种全生成式的流匹配框架,用于从单目视频中恢复时序一致的3D手部姿态与形状。给定视觉与骨骼观测,HandFlow通过一次ODE积分对整个时间窗口的MANO参数进行去噪。采用类似Flux的双流变换器,跨序列注意力捕捉长程依赖,无需自回归解码;引入置信度感知的连续掩码机制,将观测特征与可学习掩码令牌融合,有效处理噪声或缺失数据。在DexYCB和HOT3D数据集上,HandFlow达到最先进性能,世界空间误差降低超30%,加速度误差最低,且每帧重建速度达47帧/秒(单卡),较最快现有方法快约12倍,重建耗时占端到端延迟极小部分。

原文摘要 · Abstract (English)

Accurate monocular 4D hand reconstruction remains challenging. Per-frame discriminative regressors lack temporal context and often produce jittery predictions. Temporal models improve consistency by aggregating information across frames, but they are typically deterministic regressors, making them vulnerable to ambiguous observations caused by occlusion and motion blur. Generative modeling offers a natural alternative by learning a prior over plausible hand motion sequences, enabling coherent hand-state recovery when visual evidence is incomplete or unreliable. Motivated by this observation, we present HandFlow, a fully generative flow-matching framework for temporally coherent 3D hand pose and shape estimation from monocular video. Given visual and skeletal observations, HandFlow denoises an entire temporal window of MANO parameters through a single ODE integration. To support this, we use a Flux-style dual-stream transformer that attends across the full sequence to capture long-range dependencies without autoregressive decoding, and a confidence-aware continuous masking mechanism that blends observed features with learnable mask tokens to handle noisy or missing observations. Experiments on DexYCB and HOT3D show that HandFlow achieves state-of-the-art performance, with particularly large gains in world-space accuracy and temporal smoothness. It reduces world-space pose error by over 30% compared with the strongest baseline and achieves the lowest acceleration error among all evaluated methods, while remaining competitive in per-frame pose accuracy. Moreover, on a single GPU HandFlow reconstructs a 150-frame sequence at 47 fps, about 12x faster than the fastest prior video-based method, with reconstruction itself accounting for only a small fraction of the end-to-end latency.

4D手部重建流匹配生成模型单目视频

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。