让3D场景风格迁移更清晰,保留物体结构与艺术语义。
SSGaussian: Semantic-Aware and Structure-Preserving 3D Style Transfer
- 用2D扩散模型生成关键视角的风格化图像,融合多视角注意力对齐
- 实现物体级别风格一致性,使3D结果结构分明、视觉连贯
- 适合需要高质量艺术化3D重建的创作者和研究者
近年来,神经表示技术(如神经辐射场和3D高斯泼溅)推动了3D场景风格迁移的发展。现有方法虽能在3D一致的神经表示上转移风格模式,但难以有效提取并传递参考图像中的高层风格语义,且生成结果常缺乏结构清晰度与实例分离性。为此,我们提出一种新型3D风格迁移流程,融合预训练2D扩散模型的先验知识。该流程包含两个关键阶段:首先利用扩散先验生成关键视角的风格化渲染图;随后将这些风格化关键视图转移到3D表示中。过程引入两项创新设计:一是跨视角风格对齐,在UNet最后一层上采样块插入跨视角注意力,促进多视角特征交互,确保生成风格化视图兼具风格保真度与实例一致性;二是实例级风格迁移,通过利用风格化关键视图间的实例一致性,将其有效迁移到3D表示,实现更结构化、视觉连贯且富有艺术性的风格化效果。大量定性和定量实验表明,本方法在从正向到复杂360°环境的多种场景下均显著优于当前最优方法。项目页:https://jm-xu.github.io/SSGaussian 可查看沉浸式可视化。
原文摘要 · Abstract (English)
Recent advancements in neural representations, such as Neural Radiance Fields and 3D Gaussian Splatting, have increased interest in applying style transfer to 3D scenes. While existing methods can transfer style patterns onto 3D-consistent neural representations, they struggle to effectively extract and transfer high-level style semantics from the reference style image. Additionally, the stylized results often lack structural clarity and separation, making it difficult to distinguish between different instances or objects within the 3D scene. To address these limitations, we propose a novel 3D style transfer pipeline that effectively integrates prior knowledge from pretrained 2D diffusion models. Our pipeline consists of two key stages: First, we leverage diffusion priors to generate stylized renderings of key viewpoints. Then, we transfer the stylized key views onto the 3D representation. This process incorporates two innovative designs. The first is cross-view style alignment, which inserts cross-view attention into the last upsampling block of the UNet, allowing feature interactions across multiple key views. This ensures that the diffusion model generates stylized key views that maintain both style fidelity and instance-level consistency. The second is instance-level style transfer, which effectively leverages instance-level consistency across stylized key views and transfers it onto the 3D representation. This results in a more structured, visually coherent, and artistically enriched stylization. Extensive qualitative and quantitative experiments demonstrate that our 3D style transfer pipeline significantly outperforms state-of-the-art methods across a wide range of scenes, from forward-facing to challenging 360-degree environments. Visit our project page https://jm-xu.github.io/SSGaussian for immersive visualization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。