仅用几张照片就能精准分割3D物体，无需复杂标注。
ASIA: Adaptive 3D Segmentation using Few Image Annotations
- 用少量真实场景照片引导3D分割，避免繁琐的3D标注。
- 通过扩散模型迁移图像语义，实现跨几何结构的分割。
- 适合需要快速分割任意3D物体的科研与工业应用。
我们提出ASIA（基于少量图像标注的自适应3D分割），一种新框架，可对3D物体中可能非语义、不可用文本描述的“部分”进行分割。用户仅需提供少量真实场景中的图像标注，这些图像比多视角图像更易获取，比3D模型标注更省力，且比模糊的文本描述更精确。方法利用文本到图像扩散模型（如Stable Diffusion）丰富的先验知识，将图像空间的分割结果迁移到3D空间，即使标注对象与目标物体在几何或结构上差异较大也能有效工作。训练时，为每个分割区域优化一个文本标记，并使用新颖的跨视图部件对应损失进行微调。推理时，对3D网格的多视角渲染图进行分割，通过投票融合标签至UV空间，采用新型噪声优化技术精修标签，最终将UV标签映射回网格。ASIA在定量和定性评估中均显著优于现有方法，为语义与非语义3D分割任务提供了实用且可泛化的解决方案。
原文摘要 · Abstract (English)
We introduce ASIA (Adaptive 3D Segmentation using few Image Annotations), a novel framework that enables segmentation of possibly non-semantic and non-text-describable "parts" in 3D. Our segmentation is controllable through a few user-annotated in-the-wild images, which are easier to collect than multi-view images, less demanding to annotate than 3D models, and more precise than potentially ambiguous text descriptions. Our method leverages the rich priors of text-to-image diffusion models, such as Stable Diffusion (SD), to transfer segmentations from image space to 3D, even when the annotated and target objects differ significantly in geometry or structure. During training, we optimize a text token for each segment and fine-tune our model with a novel cross-view part correspondence loss. At inference, we segment multi-view renderings of the 3D mesh, fuse the labels in UV-space via voting, refine them with our novel Noise Optimization technique, and finally map the UV-labels back onto the mesh. ASIA provides a practical and generalizable solution for both semantic and non-semantic 3D segmentation tasks, outperforming existing methods by a noticeable margin in both quantitative and qualitative evaluations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。