用智能生成法重建视频中手物交互,克服遮挡与初始化失败难题。
AGILE: Hand-Object Interaction Reconstruction from Video via Agentic Generation
- 通过视觉语言模型引导生成完整纹理的物体网格,摆脱视频遮挡影响。
- 无需脆弱的SfM,用基础模型初始化并跟踪物体姿态,鲁棒性更强。
- 融合语义与物理约束优化,生成可直接用于机器人仿真的真实资产。
从单目视频重建动态手物交互对灵巧操作数据采集及机器人、虚拟现实中的数字孪生构建至关重要。但现有方法面临两大障碍:(1)依赖神经渲染常在严重遮挡下生成碎片化、不可仿真几何;(2)依赖脆弱的结构光运动(SfM)初始化,导致野外视频频繁失败。为此,我们提出AGILE框架,将范式从重建转向智能体生成。首先,采用智能体流程,由视觉语言模型(VLM)引导生成模型合成完整、封闭且高保真纹理的物体网格,不受视频遮挡影响。其次,完全绕过脆弱的SfM,提出稳健的锚定-跟踪策略:在交互起始帧使用基础模型初始化物体位姿,并利用生成资产与视频观测间的强视觉相似性进行时序传播。最后,通过接触感知优化,整合语义、几何与交互稳定性约束,确保物理合理性。在HO3D、DexYCB、ARCTIC及野外视频上的大量实验表明,AGILE在全局几何精度上优于基线,且在以往方法频繁崩溃的挑战性序列中表现出色。通过优先保证物理真实性,本方法生成的资产经真实到仿真重定向验证,可直接用于机器人应用。项目页:https://agile-hoi.github.io。
原文摘要 · Abstract (English)
Reconstructing dynamic hand-object interactions from monocular videos is critical for dexterous manipulation data collection and creating realistic digital twins for robotics and VR. However, current methods face two prohibitive barriers: (1) reliance on neural rendering often yields fragmented, non-simulation-ready geometries under heavy occlusion, and (2) dependence on brittle Structure-from-Motion (SfM) initialization leads to frequent failures on in-the-wild footage. To overcome these limitations, we introduce AGILE, a robust framework that shifts the paradigm from reconstruction to agentic generation for interaction learning. First, we employ an agentic pipeline where a Vision-Language Model (VLM) guides a generative model to synthesize a complete, watertight object mesh with high-fidelity texture, independent of video occlusions. Second, bypassing fragile SfM entirely, we propose a robust anchor-and-track strategy. We initialize the object pose at a single interaction onset frame using a foundation model and propagate it temporally by leveraging the strong visual similarity between our generated asset and video observations. Finally, a contact-aware optimization integrates semantic, geometric, and interaction stability constraints to enforce physical plausibility. Extensive experiments on HO3D, DexYCB, ARCTIC, and in-the-wild videos reveal that AGILE outperforms baselines in global geometric accuracy while demonstrating exceptional robustness on challenging sequences where prior arts frequently collapse. By prioritizing physical validity, our method produces simulation-ready assets validated via real-to-sim retargeting for robotic applications. Project page: https://agile-hoi.github.io.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。