arXiv:2603.11755cs.CV2026-03被引 4

用稀疏3D手部关节点控制,生成更真实的手物交互视频。

Controllable Egocentric Video Generation via Occlusion-Aware Sparse 3D Hand Joints

  • 以稀疏3D手关节为显式控制信号,提升动作一致性
  • 在100万条视频上训练,生成视频手部动作更逼真
  • 适合机器人手控、虚拟交互等需精细控制场景

可控视频生成在复杂手物交互中对构建视觉世界模型至关重要。现有方法常因依赖密集2D轨迹或隐式姿态表示,导致几何结构模糊,出现运动不一致与遮挡幻觉。为此,我们提出利用稀疏3D手关节作为显式控制信号,具备显式几何建模、直观交互编辑及跨体泛化能力。设计高效控制模块,通过惩罚被遮挡关节的不可靠特征,并采用基于3D的加权机制处理动态遮挡目标关节。同时将3D几何嵌入直接注入潜在空间以保持结构一致性。为支持稳健训练与评估,构建自动化标注流程,生成100万条高质量头戴视角视频片段及精确手部轨迹。实验表明,该方法优于当前最优基线,生成高保真头戴视角视频,实现逼真的手物交互。

原文摘要 · Abstract (English)

Controllable video generation for complex hand-object interactions is a critical step toward building visual world models. However, existing methods often struggle to achieve fine-grained, 3D-consistent hand articulation in generated videos. By relying on dense 2D trajectories or implicit pose representations, they collapse crucial geometric structures into spatially ambiguous signals, leading to severe motion inconsistencies and hallucinated artifacts under egocentric occlusions. To address this, we propose leveraging sparse 3D hand joints as explicit control signals with three key advantages: explicit geometry to resolve occlusions, an intuitive interface for interactive editing, and cross-embodiment generalization to robotic hands. Built upon this, our efficient control module extracts occlusion-aware features from the source reference frame by penalizing unreliable visual features from hidden joints, and employs a 3D-based weighting mechanism to handle dynamically occluded target joints during motion propagation. Meanwhile, it directly injects 3D geometric embeddings into the latent space to enforce structural consistency. To facilitate robust training and evaluation, we develop an automated annotation pipeline, yielding 1M high-quality egocentric video clips paired with precise hand trajectories. Experiments demonstrate that our approach outperforms state-of-the-art baselines, generating high-fidelity egocentric videos with realistic hand-object interactions.

视频生成3D控制手部交互扩散模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。