用文本指令快速精准编辑3D网格,保持原始形状同时聚焦关键区域。
LEMON: Localized Editing with Mesh Optimization and Neural Shaders
- 结合神经延迟着色与局部网格优化,聚焦重要顶点进行编辑。
- 在DTU数据集上比当前最佳方法更快生成精细编辑的网格。
- 适合需要快速3D内容修改的工业设计与动画制作场景。
在实际应用中,对多边形网格进行编辑比生成新网格更快,但仍可能耗时且困难。现有方法通常只关注几何或新视图合成单一任务,导致网格与视图结果不一致。本文提出LEMON,一种将神经延迟着色与局部网格优化结合的网格编辑流程。该方法首先利用分割模型识别网格中最重要的顶点区域,再基于多视角图像同时优化神经着色器与多边形网格,并从每视角提取法线图与渲染图像作为条件。随后,使用文生图扩散模型对输入图像进行编辑,并迭代更新数据集以变形网格。最终生成符合文本指令的编辑后网格,保留初始网格的几何特征并聚焦显著区域。我们在DTU数据集上评估了该流程,结果表明其生成精细编辑网格的速度优于当前最先进方法。代码与附加结果见补充材料。
原文摘要 · Abstract (English)
In practical use cases, polygonal mesh editing can be faster than generating new ones, but it can still be challenging and time-consuming for users. Existing solutions for this problem tend to focus on a single task, either geometry or novel view synthesis, which often leads to disjointed results between the mesh and view. In this work, we propose LEMON, a mesh editing pipeline that combines neural deferred shading with localized mesh optimization. Our approach begins by identifying the most important vertices in the mesh for editing, utilizing a segmentation model to focus on these key regions. Given multi-view images of an object, we optimize a neural shader and a polygonal mesh while extracting the normal map and the rendered image from each view. By using these outputs as conditioning data, we edit the input images with a text-to-image diffusion model and iteratively update our dataset while deforming the mesh. This process results in a polygonal mesh that is edited according to the given text instruction, preserving the geometric characteristics of the initial mesh while focusing on the most significant areas. We evaluate our pipeline using the DTU dataset, demonstrating that it generates finely-edited meshes more rapidly than the current state-of-the-art methods. We include our code and additional results in the supplementary material.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。