无需修改主模型,用轻量适配器实现高质量多视角图像生成。
MV-Adapter: Multi-view Consistent Image Generation Made Easy
- 设计可插拔适配器,仅更新少量参数提升文本到图像模型的多视角能力。
- 在SDXL上实现768分辨率多视角生成,图像质量优于现有方法。
- 支持任意视角生成,适合需要高效3D内容创作的开发者与研究者。
现有多视角图像生成方法通常对预训练文本到图像(T2I)模型进行深度修改并需全量微调,导致计算成本高(尤其在大模型和高分辨率下),且因优化困难与高质量3D数据稀缺,易造成图像质量下降。本文提出首个基于适配器的多视角图像生成方案——MV-Adapter,一种无需改变原网络结构或特征空间的即插即用组件。通过仅更新少量参数,MV-Adapter实现高效训练,保留预训练模型的先验知识,降低过拟合风险。为高效建模3D几何知识,引入重复自注意力层与并行注意力架构,使适配器能继承预训练模型的强大先验以学习新3D信息。此外,提出统一条件编码器,无缝融合相机参数与几何信息,支持基于文本或图像的3D生成及贴图应用。MV-Adapter在Stable Diffusion XL(SDXL)上实现768分辨率多视角生成,展现良好适应性与通用性,并可扩展至任意视角生成,显著提升生成质量与效率,树立新标准。
原文摘要 · Abstract (English)
Existing multi-view image generation methods often make invasive modifications to pre-trained text-to-image (T2I) models and require full fine-tuning, leading to (1) high computational costs, especially with large base models and high-resolution images, and (2) degradation in image quality due to optimization difficulties and scarce high-quality 3D data. In this paper, we propose the first adapter-based solution for multi-view image generation, and introduce MV-Adapter, a versatile plug-and-play adapter that enhances T2I models and their derivatives without altering the original network structure or feature space. By updating fewer parameters, MV-Adapter enables efficient training and preserves the prior knowledge embedded in pre-trained models, mitigating overfitting risks. To efficiently model the 3D geometric knowledge within the adapter, we introduce innovative designs that include duplicated self-attention layers and parallel attention architecture, enabling the adapter to inherit the powerful priors of the pre-trained models to model the novel 3D knowledge. Moreover, we present a unified condition encoder that seamlessly integrates camera parameters and geometric information, facilitating applications such as text- and image-based 3D generation and texturing. MV-Adapter achieves multi-view generation at 768 resolution on Stable Diffusion XL (SDXL), and demonstrates adaptability and versatility. It can also be extended to arbitrary view generation, enabling broader applications. We demonstrate that MV-Adapter sets a new quality standard for multi-view image generation, and opens up new possibilities due to its efficiency, adaptability and versatility.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。