arXiv:2602.06226cs.CV2026-02被引 4

一分钟内从日常手物交互视频重建3D物体,速度超快且抗遮挡。

ForeHOI: Feed-forward 3D Object Reconstruction from Daily Hand-Object Interaction Videos

  • 前馈框架联合预测2D掩码与3D形状,直接处理严重遮挡问题。
  • 在真实数据上达到顶尖性能,比优化方法快约100倍。
  • 适合做手势交互、机器人抓取等需要快速3D重建的场景。

日常单目手物交互视频广泛存在,是具身智能的重要资源。尽管单目视频中手部三维重建已取得显著进展,但受严重遮挡和相机、手、物体耦合运动影响,物体重建仍具挑战。本文提出ForeHOI,一种前馈模型,可在一分钟内直接从单目手物交互视频重建3D物体几何,无需任何预处理。核心思路是在前馈框架中联合预测2D掩码修复与3D形状补全,有效应对单目手持物体视频中的严重遮挡,性能超越优化类方法。2D与3D形状补全间的信息交互提升了整体重建质量,使系统能有效处理严重遮挡。此外,为支持模型训练,我们构建了首个大规模高保真合成手物交互数据集,含完整标注。大量实验表明,ForeHOI在物体重建上达到当前最优表现,相比以往方法提速约100倍。代码与数据见:https://github.com/Tao-11-chen/ForeHOI。

原文摘要 · Abstract (English)

The ubiquity of monocular videos capturing daily hand-object interactions presents a valuable resource for embodied intelligence. While 3D hand reconstruction from in-the-wild videos has seen significant progress, reconstructing the involved objects remains challenging due to severe occlusions and the complex, coupled motion of the camera, hands, and object. In this paper, we introduce ForeHOI, a novel feed-forward model that directly reconstructs 3D object geometry from monocular hand-object interaction videos within one minute of inference time, eliminating the need for any pre-processing steps. Our key insight is that, the joint prediction of 2D mask inpainting and 3D shape completion in a feed-forward framework can effectively address the problem of severe occlusion in monocular hand-held object videos, thereby achieving results that outperform the performance of optimization-based methods. The information exchanges between the 2D and 3D shape completion boosts the overall reconstruction quality, enabling the framework to effectively handle severe hand-object occlusion. Furthermore, to support the training of our model, we contribute the first large-scale, high-fidelity synthetic dataset of hand-object interactions with comprehensive annotations. Extensive experiments demonstrate that ForeHOI achieves state-of-the-art performance in object reconstruction, significantly outperforming previous methods with around a 100x speedup. Code and data are available at: https://github.com/Tao-11-chen/ForeHOI.

3D重建手物交互前馈模型实时重建

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。