arXiv:2504.05303cs.CV2025-04CVPR被引 24

用2D大模型推3D人体物体重合点,无需昂贵标注。

InteractVLM: 3D Interaction Reasoning from 2D Foundational Models

  • 用多视角渲染把3D物体映射到2D,再定位接触点后还原3D
  • 在真实图片上实现高精度3D人体与物体交互重建
  • 适合做3D交互分析、虚拟试衣等需要真实场景建模的领域

我们提出InteractVLM,一种从单张自然图像中估计人体与物体3D接触点的新方法,实现精准的人体-物体联合3D重建。该任务因遮挡、深度模糊及物体形状差异大而极具挑战。现有方法依赖昂贵的动作捕捉系统或人工标注的3D接触数据,难以扩展且泛化能力差。为此,InteractVLM利用大规模视觉语言模型(VLMs)的广泛视觉知识,并以少量3D接触数据进行微调。由于原始模型仅处理2D信息,而接触本质是3D,我们设计了全新的Render-Localize-Lift模块:(1) 通过多视角渲染将3D表面嵌入2D空间;(2) 训练新型多视图定位模型(MV-Loc)在2D中预测接触点;(3) 将2D结果提升至3D。此外,我们提出新任务——语义人体接触估计,让接触预测显式依赖物体语义,实现更丰富的交互建模。InteractVLM在接触估计上优于现有方法,并可从自然图像完成3D重建。代码与模型见https://interactvlm.is.tue.mpg.de。

原文摘要 · Abstract (English)

We introduce InteractVLM, a novel method to estimate 3D contact points on human bodies and objects from single in-the-wild images, enabling accurate human-object joint reconstruction in 3D. This is challenging due to occlusions, depth ambiguities, and widely varying object shapes. Existing methods rely on 3D contact annotations collected via expensive motion-capture systems or tedious manual labeling, limiting scalability and generalization. To overcome this, InteractVLM harnesses the broad visual knowledge of large Vision-Language Models (VLMs), fine-tuned with limited 3D contact data. However, directly applying these models is non-trivial, as they reason only in 2D, while human-object contact is inherently 3D. Thus we introduce a novel Render-Localize-Lift module that: (1) embeds 3D body and object surfaces in 2D space via multi-view rendering, (2) trains a novel multi-view localization model (MV-Loc) to infer contacts in 2D, and (3) lifts these to 3D. Additionally, we propose a new task called Semantic Human Contact estimation, where human contact predictions are conditioned explicitly on object semantics, enabling richer interaction modeling. InteractVLM outperforms existing work on contact estimation and also facilitates 3D reconstruction from an in-the wild image. Code and models are available at https://interactvlm.is.tue.mpg.de.

3D重建交互理解视觉语言模型接触估计

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。