从单张图生成可交互的关节物体,支持精确分部与运动模拟。
DreamArt: Generating Interactable Articulated Objects from a Single Image
- 分三阶段生成:先分割重建,再用视频扩散模型学关节动作,最后优化运动与纹理。
- 生成物体零件准确、外观真实,关节运动自然且无遮挡模糊。
- 适合做智能体交互、虚拟现实中的可动物体建模,无需多视角数据。
生成可动物体(如笔记本电脑、微波炉)是具身智能与AR/VR中关键但极具挑战的任务。现有图像转3D方法多关注表面几何与纹理,忽略部件分解与关节建模;而神经重建方法(如NeRF或Gaussian Splatting)依赖密集多视角或交互数据,扩展性差。本文提出DreamArt,一种从单视角图像生成高保真、可交互关节物体的新框架。该框架采用三阶段流程:首先通过图像转3D生成、掩码提示3D分割与部件非可视补全,重建出带部件分割的完整3D网格;其次,微调视频扩散模型以捕捉部件级运动先验,使用可动部件掩码与非可视图像作为提示,缓解遮挡带来的歧义;最后,以双四元数表示关节运动,进行全局纹理精修与重绘,确保各部件间纹理连贯性与高质量。实验表明,DreamArt能有效生成具有准确部件形状、高外观保真度及合理关节运动的物体,为可动资产生成提供可扩展方案。
原文摘要 · Abstract (English)
Generating articulated objects, such as laptops and microwaves, is a crucial yet challenging task with extensive applications in Embodied AI and AR/VR. Current image-to-3D methods primarily focus on surface geometry and texture, neglecting part decomposition and articulation modeling. Meanwhile, neural reconstruction approaches (e.g., NeRF or Gaussian Splatting) rely on dense multi-view or interaction data, limiting their scalability. In this paper, we introduce DreamArt, a novel framework for generating high-fidelity, interactable articulated assets from single-view images. DreamArt employs a three-stage pipeline: firstly, it reconstructs part-segmented and complete 3D object meshes through a combination of image-to-3D generation, mask-prompted 3D segmentation, and part amodal completion. Second, we fine-tune a video diffusion model to capture part-level articulation priors, leveraging movable part masks as prompt and amodal images to mitigate ambiguities caused by occlusion. Finally, DreamArt optimizes the articulation motion, represented by a dual quaternion, and conducts global texture refinement and repainting to ensure coherent, high-quality textures across all parts. Experimental results demonstrate that DreamArt effectively generates high-quality articulated objects, possessing accurate part shape, high appearance fidelity, and plausible articulation, thereby providing a scalable solution for articulated asset generation. Our project page is available at https://dream-art-0.github.io/DreamArt/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。