让3D场景自动匹配多种艺术风格,同时保持高效训练。
Multi-StyleGS: Stylizing Gaussian Splatting with Multiple Styles
- 通过双向匹配自动识别风格图与场景局部对应关系
- 引入语义风格损失和局部全局特征匹配,提升多视角一致性
- 支持灵活编辑,内存效率高,细节更丰富
近年来,为满足创作需求,人们越来越希望将给定的3D场景风格化,使其与参考图像的艺术风格一致。尽管3D高斯点云(Gaussian Splatting, GS)已成为一种高效且逼真的3D建模方法,但如何在保持内存效率的前提下,通过自动局部风格迁移或手动指定,实现多风格适配仍面临挑战。本文提出一种名为Multi-StyleGS的新方法,采用双向匹配机制自动建立风格图像与渲染图像局部区域之间的对应关系。为支持局部风格迁移,提出一种基于分割网络的语义风格损失函数,可对场景中不同物体施加各异风格,并引入局部-全局特征匹配以增强多视角一致性。此外,该方法具备高效训练、更丰富的纹理细节和更优的颜色匹配能力。为提升每个高斯点的语义标签鲁棒性,还设计了多项分割网络正则化技术。大量实验证明,该方法在生成合理风格化结果及提供灵活编辑方面优于现有方法。
原文摘要 · Abstract (English)
In recent years, there has been a growing demand to stylize a given 3D scene to align with the artistic style of reference images for creative purposes. While 3D Gaussian Splatting(GS) has emerged as a promising and efficient method for realistic 3D scene modeling, there remains a challenge in adapting it to stylize 3D GS to match with multiple styles through automatic local style transfer or manual designation, while maintaining memory efficiency for stylization training. In this paper, we introduce a novel 3D GS stylization solution termed Multi-StyleGS to tackle these challenges. In particular, we employ a bipartite matching mechanism to au tomatically identify correspondences between the style images and the local regions of the rendered images. To facilitate local style transfer, we introduce a novel semantic style loss function that employs a segmentation network to apply distinct styles to various objects of the scene and propose a local-global feature matching to enhance the multi-view consistency. Furthermore, this technique can achieve memory efficient training, more texture details and better color match. To better assign a robust semantic label to each Gaussian, we propose several techniques to regularize the segmentation network. As demonstrated by our comprehensive experiments, our approach outperforms existing ones in producing plausible stylization results and offering flexible editing.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。