不用训练,用生成图像模型直接提取3D形状的几何基元。
Harnessing Generative Image Models for Training-Free Primitive Shape Abstraction
- 用多视角图像+视觉语言模型识别语义部分,生成色标分割图。
- 平均每物体用5-9个超二次曲面基元,Chamfer距离最优。
- 零参数设计,不依赖类别和方向,适合通用3D理解任务。
将3D形状表示为紧凑的几何基元集是机器人、模拟与场景理解的基础。大规模训练的生成图像模型已成为通用视觉学习者,能直接在图像域中识别并分割任意类别的物体部件,无需特定任务训练。我们提出一种无需训练的直接利用方式:渲染3D物体的多视角图像,通过视觉语言模型分析其语义部件,提示生成式图像模型绘制颜色编码的部件分割掩码,再将其重投影到几何体上,并通过参数优化为每个部件拟合一个超二次曲面基元。该方法无任何可学习参数,具备类别无关性和方向不变性,克服了以往学习型模型的局限。其性能上限随未来生成模型提升而上升,地面真值分割研究证实当前准确率瓶颈在于部件分割而非基元拟合。在HumanPrim和Toys4K数据集上,本方法以平均每物体5–9个基元达到最低的Chamfer距离。
原文摘要 · Abstract (English)
Representing 3D shapes as compact sets of geometric primitives is fundamental to robotics, simulation, and scene understanding. Generative image models trained at scale have recently emerged as generalist visual learners that can identify and segment object parts directly in the image domain, across arbitrary categories and without task-specific training. Adapting such models to downstream tasks typically requires fine-tuning; we ask whether their pretrained capability can instead be harnessed directly, without any training, and answer affirmatively with a training-free harness. Our pipeline renders multi-view images of a 3D object, uses a vision-language model to analyze its semantic parts, prompts a generative image model to paint a color-coded part segmentation mask, reprojects it onto the geometry, and fits a superquadric primitive to each part via parameter optimization. The approach contains no learned parameters: it is category-agnostic and orientation-invariant, properties that previous learning-based models struggled with. Its accuracy ceiling rises with future generative-model improvements, which we confirm with a ground-truth segmentation study showing that part segmentation, not primitive fitting, is the current accuracy bottleneck. On HumanPrim and Toys4K, our method achieves the lowest Chamfer distance among all evaluated methods, using 5--9 primitives per object on average.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。