arXiv:2412.18608cs.CV2024-12CVPR被引 65

让3D生成具备可拆分部件,支持文本/图像/原始3D输入

PartGen: Part-level 3D Generation and Reconstruction with Multi-View Diffusion Models

  • 用多视角扩散模型先分割出合理部件
  • 通过上下文补全缺失部分,实现高保真重建
  • 适合需要独立编辑部件的3D创作场景

文本或图像到3D的生成器和3D扫描仪如今能生成高质量形状与纹理的3D资产,但这些资产通常以单一融合表示(如隐式神经场、高斯混合或网格),缺乏有意义的结构。而大多数应用与创意工作流需要由多个可独立操作的部件组成的资产。为此,我们提出PartGen,一种从文本、图像或无结构3D对象出发,生成包含有意义部件的3D物体的新方法。首先,给定一个3D物体的多视角图像(生成或渲染),一个多视角扩散模型提取一组合理且视图一致的部件分割,将物体划分为若干部件。随后,第二个多视角扩散模型分别处理每个部件,填补遮挡区域,并利用这些完成的视角输入3D重建网络进行重建。该补全过程考虑整个物体的上下文,确保部件间协调融合。生成式补全模型可弥补因遮挡缺失的信息;在极端情况下,甚至可根据输入3D资产‘幻觉’出完全不可见的部件。我们在生成和真实的3D资产上评估该方法,结果表明其显著优于分割与部件提取基线。我们还展示了下游应用,如3D部件编辑。

原文摘要 · Abstract (English)

Text- or image-to-3D generators and 3D scanners can now produce 3D assets with high-quality shapes and textures. These assets typically consist of a single, fused representation, like an implicit neural field, a Gaussian mixture, or a mesh, without any useful structure. However, most applications and creative workflows require assets to be made of several meaningful parts that can be manipulated independently. To address this gap, we introduce PartGen, a novel approach that generates 3D objects composed of meaningful parts starting from text, an image, or an unstructured 3D object. First, given multiple views of a 3D object, generated or rendered, a multi-view diffusion model extracts a set of plausible and view-consistent part segmentations, dividing the object into parts. Then, a second multi-view diffusion model takes each part separately, fills in the occlusions, and uses those completed views for 3D reconstruction by feeding them to a 3D reconstruction network. This completion process considers the context of the entire object to ensure that the parts integrate cohesively. The generative completion model can make up for the information missing due to occlusions; in extreme cases, it can hallucinate entirely invisible parts based on the input 3D asset. We evaluate our method on generated and real 3D assets and show that it outperforms segmentation and part-extraction baselines by a large margin. We also showcase downstream applications such as 3D part editing.

3D生成扩散模型部件分割多视角重建

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。