从单目视频重建人手操作可动物体的4D动态,突破传统方法依赖多视角或预扫描的限制
ArtHOI: Taming Foundation Models for Monocular 4D Reconstruction of Hand-Articulated-Object Interactions
- 融合多模态大模型先验,通过自适应采样优化物体尺度与姿态
- 利用大语言模型推理接触关系,指导手物网格对齐与动态重构
- 提出新数据集,支持复杂交互场景下4D重建的全面评估
现有手物交互(HOI)方法主要针对刚性物体,而可动物体的4D重建通常需要预先扫描物体或多视角视频。从单目RGB视频重建人类-可动物体交互的4D动态仍是一个未被充分探索但极具意义的挑战。得益于基础模型的最新进展,我们提出了ArtHOI——一种基于优化的框架,整合并精炼多个基础模型的先验信息。核心贡献包括一套新方法,用于解决这些先验固有的不准确性和物理不合理性。具体地,提出自适应采样精化(ASR)方法,优化物体的度量尺度与姿态,以将归一化网格准确定位到世界空间;同时设计多模态大语言模型(MLLM)引导的手物对齐方法,利用接触推理信息作为约束,优化手物网格组合。为支持全面评估,我们还构建了两个新数据集:ArtHOI-RGBD和ArtHOI-Wild。大量实验验证了ArtHOI在多样化物体与交互下的鲁棒性与有效性。
原文摘要 · Abstract (English)
Existing hand-object interactions (HOI) methods are largely limited to rigid objects, while 4D reconstruction methods of articulated objects generally require pre-scanning the object or even multi-view videos. It remains an unexplored but significant challenge to reconstruct 4D human-articulated-object interactions from a single monocular RGB video. Fortunately, recent advancements in foundation models present a new opportunity to address this highly ill-posed problem. To this end, we introduce ArtHOI, an optimization-based framework that integrates and refines priors from multiple foundation models. Our key contribution is a suite of novel methodologies designed to resolve the inherent inaccuracies and physical unreality of these priors. In particular, we introduce an Adaptive Sampling Refinement (ASR) method to optimize object's metric scale and pose for grounding its normalized mesh in world space. Furthermore, we propose a Multimodal Large Language Model (MLLM) guided hand-object alignment method, utilizing contact reasoning information as constraints of hand-object mesh composition optimization. To facilitate a comprehensive evaluation, we also contribute two new datasets, ArtHOI-RGBD and ArtHOI-Wild. Extensive experiments validate the robustness and effectiveness of our ArtHOI across diverse objects and interactions. Project: https://arthoi-reconstruction.github.io.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。