用单步扩散模型实现3D场景多视角风格迁移,保持结构一致性和视觉质量。
Towards Multi-View Consistent Style Transfer with One-Step Diffusion via Vision Conditioning
- 基于预训练单步扩散模型,引入视觉条件模块提取风格信息。
- 在小数据集上通过LoRA适配,结合颜色与结构损失,提升多视角一致性。
- 无需3D信息即可生成高质量、低畸变的多视角风格化图像,适合影视与游戏设计。
3D场景风格化在三维视觉中日益受到关注。尽管2D风格迁移已有显著成果,直接应用于3D场景常导致结构和多视角一致性破坏,引发视点间图像扭曲。为此,本文提出OSDiffST方法,基于预训练的单步扩散模型(SD-Turbo)实现多视角图像风格迁移。为在小数据集上高效适配,引入视觉条件模块从参考风格图提取风格信息作为扩散模型的条件输入,并采用LoRA进行模型微调。同时,设计颜色分布对齐与结构相似性损失函数,以增强风格化图像与内容图像的一致性。实验表明,该方法在多种风格生成任务中优于现有方法,生成的多视角图像具备更优的结构完整性和更少畸变,且不依赖任何3D信息。代码已开源:https://github.com/YushenZuo/OSDiffST。
原文摘要 · Abstract (English)
The stylization of 3D scenes is an increasingly attractive topic in 3D vision. Although image style transfer has been extensively researched with promising results, directly applying 2D style transfer methods to 3D scenes often fails to preserve the structural and multi-view properties of 3D environments, resulting in unpleasant distortions in images from different viewpoints. To address these issues, we leverage the remarkable generative prior of diffusion-based models and propose a novel style transfer method, OSDiffST, based on a pre-trained one-step diffusion model (i.e., SD-Turbo) for rendering diverse styles in multi-view images of 3D scenes. To efficiently adapt the pre-trained model for multi-view style transfer on small datasets, we introduce a vision condition module to extract style information from the reference style image to serve as conditional input for the diffusion model and employ LoRA in diffusion model for adaptation. Additionally, we consider color distribution alignment and structural similarity between the stylized and content images using two specific loss functions. As a result, our method effectively preserves the structural information and multi-view consistency in stylized images without any 3D information. Experiments show that our method surpasses other promising style transfer methods in synthesizing various styles for multi-view images of 3D scenes. Stylized images from different viewpoints generated by our method achieve superior visual quality, with better structural integrity and less distortion. The source code is available at https://github.com/YushenZuo/OSDiffST.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。