用注意力机制精准定位3D高斯点,实现文本驱动的精细编辑
GSEditPro: 3D Gaussian Splatting Editing with Attention-based Progressive Localization
- 通过交叉注意力提取文本提示语义,逐级定位高斯点
- 结合分数蒸馏采样与伪真值优化,提升编辑稳定性与细节质量
- 适合需要精确控制3D场景编辑的设计师与研究人员
随着大规模文本到图像(T2I)模型和隐式3D表示如神经辐射场(NeRF)的发展,基于NeRF的文本驱动生成编辑方法不断涌现。然而,几何与纹理信息的隐式编码导致编辑过程中对象定位与控制不够准确。近期,3D高斯泼溅(3D Gaussian Splatting)这一依赖显式表示的实时渲染技术在编辑方法上取得进展,但仍存在定位不准、操控能力有限等问题。为此,我们提出GSEditPro,一种仅需文本提示即可实现多样且精准编辑的3D场景编辑框架。利用3D高斯分布的显式特性,引入基于注意力的渐进式定位模块,在渲染过程中为每个高斯点添加语义标签。该模块通过T2I模型的交叉注意力层提取编辑提示的相关性,实现对编辑区域的精准定位。此外,我们提出一种基于3D高斯泼溅的创新编辑优化方法,借助分数蒸馏采样(Score Distillation Sampling)与伪真值引导,获得稳定且精细化的编辑结果。大量实验验证了方法的有效性。
原文摘要 · Abstract (English)
With the emergence of large-scale Text-to-Image(T2I) models and implicit 3D representations like Neural Radiance Fields (NeRF), many text-driven generative editing methods based on NeRF have appeared. However, the implicit encoding of geometric and textural information poses challenges in accurately locating and controlling objects during editing. Recently, significant advancements have been made in the editing methods of 3D Gaussian Splatting, a real-time rendering technology that relies on explicit representation. However, these methods still suffer from issues including inaccurate localization and limited manipulation over editing. To tackle these challenges, we propose GSEditPro, a novel 3D scene editing framework which allows users to perform various creative and precise editing using text prompts only. Leveraging the explicit nature of the 3D Gaussian distribution, we introduce an attention-based progressive localization module to add semantic labels to each Gaussian during rendering. This enables precise localization on editing areas by classifying Gaussians based on their relevance to the editing prompts derived from cross-attention layers of the T2I model. Furthermore, we present an innovative editing optimization method based on 3D Gaussian Splatting, obtaining stable and refined editing results through the guidance of Score Distillation Sampling and pseudo ground truth. We prove the efficacy of our method through extensive experiments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。