用大模型从单张图重建手物交互，效果远超传统方法。
EasyHOI: Unleashing the Power of Large Models for Reconstructing Hand-Object Interactions in the Wild
- 借助现成大模型估计手姿与物体形状
- 通过前后景约束优化，提升重建精度
- 适合需要真实场景手物交互的应用
本工作旨在从单张图像中重建手物交互，该任务基础但病态。与基于视频、多视角图像或预定义3D模板的方法不同，单视图重建因固有歧义和遮挡面临巨大挑战，且手姿多样、物体形状尺寸各异进一步加剧难度。我们的核心洞察是：当前用于分割、修复和3D重建的通用模型在真实图像上具有强泛化能力，可提供有力的视觉与几何先验。具体而言，给定单张图像，我们设计了一种新流水线，利用现成大模型估计底层手姿与物体形状；随后，基于初始重建结果，采用先验引导的优化方案，使手姿满足3D物理约束并匹配2D输入内容。我们在多个数据集上进行实验，结果表明该方法持续优于基线，能忠实还原各类手物交互。项目主页：https://lym29.github.io/EasyHOI-page/
原文摘要 · Abstract (English)
Our work aims to reconstruct hand-object interactions from a single-view image, which is a fundamental but ill-posed task. Unlike methods that reconstruct from videos, multi-view images, or predefined 3D templates, single-view reconstruction faces significant challenges due to inherent ambiguities and occlusions. These challenges are further amplified by the diverse nature of hand poses and the vast variety of object shapes and sizes. Our key insight is that current foundational models for segmentation, inpainting, and 3D reconstruction robustly generalize to in-the-wild images, which could provide strong visual and geometric priors for reconstructing hand-object interactions. Specifically, given a single image, we first design a novel pipeline to estimate the underlying hand pose and object shape using off-the-shelf large models. Furthermore, with the initial reconstruction, we employ a prior-guided optimization scheme, which optimizes hand pose to comply with 3D physical constraints and the 2D input image content. We perform experiments across several datasets and show that our method consistently outperforms baselines and faithfully reconstructs a diverse set of hand-object interactions. Here is the link of our project page: https://lym29.github.io/EasyHOI-page/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。