用视频扩散模型重建双手动作,无需检测器也能处理遮挡。
The Surprising Effectiveness of Video Diffusion Models for Hand Motion Reconstruction

- 用预训练视频扩散模型提取手部特征,通过手部渲染目标微调
- 在ARCTIC、HOT3D等数据集上显著优于现有方法
- 适合需要真实场景手部动作数据的具身AI研究
从第一人称视频中进行4D手部动作重建面临现有方法的瓶颈:基于图像的方法依赖易受遮挡影响的检测器;基于视频的方法则仅依赖稀疏的手部姿态标注,无法充分学习运动动态、遮挡推理和手物交互。而视频生成模型在互联网规模视频上训练时,必须隐式掌握这些能力。我们提出ViDiHand,利用预训练视频扩散模型的表征进行双手姿态重建。通过手部叠加渲染目标对模型进行微调,使其专用于手部特征提取,同时保留世界先验。解码器从微调后的特征中恢复出度量尺度的姿态。整个流程直接作用于完整帧,无需检测器、补全模块或测试时优化。在ARCTIC、HOT3D和HOI4D数据集上,ViDiHand显著优于先前方法,证明视频扩散模型是手部动作重建的强大新基础,并为具身AI的规模化野外数据采集提供了可行路径。
原文摘要 · Abstract (English)
4D hand motion reconstruction from egocentric video is bottlenecked by clear limitations of existing methods: image-based pipelines depend on a detector that fails under heavy occlusion, while video-based methods rely on temporal modules learned only from scarce hand-pose annotations, a narrow signal insufficient to model motion dynamics, occlusion reasoning, and hand-object interaction. These capabilities, however, are exactly what video generative models must implicitly acquire when trained to synthesize coherent video at internet scale. Motivated by this, we present ViDiHand, which leverages the representations of a pretrained video diffusion model to reconstruct 4D two-hand pose. We adapt it via a hand-overlay rendering objective that specializes its features for hands while preserving its world priors. A decoder then recovers metric-scale pose from the adapted features. The whole pipeline operates directly on full frames--no detector, no infiller, and no test-time optimization. On ARCTIC, HOT3D, and HOI4D, ViDiHand substantially outperforms prior methods, establishing video diffusion models as a powerful new foundation for hand motion reconstruction and a promising route to scalable in-the-wild data collection for embodied AI. Project page: https://vidihand.github.io.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。