arXiv:2411.19322cs.CVcs.GR2024-11被引 11

无需优化,秒级完成3D物体材质选择与分割

SAMa: Material-aware 3D Selection and Segmentation

  • 基于视频先验构建材质视角数据集,实现2D预测向3D投影
  • 通过点云近邻查找,秒级生成多视角一致的材质掩码
  • 适合3D内容创作、材质替换与编辑,支持任意3D表示

将3D资产按材质分解是艺术家常用但高度手动的任务。本文提出面向任意3D表示中真实场景物体的材质选择方法SAMa。基于SAM2的视频先验,我们构建了面向材质领域的视频数据集。通过深度图将每张视图投影至中间3D点云,利用该点云与任意3D表示间的最近邻查找,高效重建准确的表面选择掩码,可从任意视角查看。方法设计保证多视角一致性,无需耗时的逐资产优化,在数秒内完成无优化选择。SAMa在选择精度和多视角一致性上优于多个强基线,支持多种应用,如将文本生成3D结果中的漫反射材质替换为PBR材质,或对NeRF和3DGS捕获结果进行材质选择与编辑。

原文摘要 · Abstract (English)

Decomposing 3D assets into material parts is a common task for artists, yet remains a highly manual process. In this work, we introduce Select Any Material (SAMa), a material selection approach for in-the-wild objects in arbitrary 3D representations. Building on SAM2's video prior, we construct a material-centric video dataset that extends it to the material domain. We propose an efficient way to lift the model's 2D predictions to 3D by projecting each view into an intermediary 3D point cloud using depth. Nearest-neighbor lookups between any 3D representation and this similarity point cloud allow us to efficiently reconstruct accurate selection masks over objects' surfaces that can be inspected from any view. Our method is multiview-consistent by design, alleviating the need for costly per-asset optimization, and performs optimization-free selection in seconds. SAMa outperforms several strong baselines in selection accuracy and multiview consistency and enables various compelling applications, such as replacing the diffuse-textured materials on a text-to-3D output with PBR materials or selecting and editing materials on NeRFs and 3DGS captures.

3D分割材质识别无优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。