arXiv:2412.20800cs.CV2024-12被引 9

通过控制交叉注意力提升文生图模型美感,无需重训练即可增强视觉质量。

VMix: Improving Text-to-Image Diffusion Model with Cross-Attention Mixing Control

  • 将文本提示拆分为内容与美学描述,用嵌入初始化实现解耦
  • 在去噪过程中通过值混合交叉注意力注入美学条件,提升图像质感
  • 可插即用,兼容LoRA、ControlNet等模块,适合追求高质量生成的用户

尽管扩散模型在文生图任务中表现卓越,仍难以生成具有高度美学品质的图像,尤其在色彩、光影、构图等细粒度维度上与真实美学图像存在差距。本文提出跨注意力值混合控制(VMix)适配器,一种即插即用的美学增强模块。其核心思路是:(1) 通过美学嵌入初始化,将输入文本提示解耦为内容描述与美学描述;(2) 利用零初始化线性层连接网络,通过值混合的交叉注意力将美学条件融入去噪过程,从而在不破坏图文对齐的前提下提升图像美学表现。该设计使VMix具备灵活性,可直接应用于社区主流模型以提升视觉性能,且无需重新训练。大量实验表明,VMix优于现有最先进方法,并可与LoRA、ControlNet、IPAdapter等模块兼容,显著改善生成图像的美学质量。

原文摘要 · Abstract (English)

While diffusion models show extraordinary talents in text-to-image generation, they may still fail to generate highly aesthetic images. More specifically, there is still a gap between the generated images and the real-world aesthetic images in finer-grained dimensions including color, lighting, composition, etc. In this paper, we propose Cross-Attention Value Mixing Control (VMix) Adapter, a plug-and-play aesthetics adapter, to upgrade the quality of generated images while maintaining generality across visual concepts by (1) disentangling the input text prompt into the content description and aesthetic description by the initialization of aesthetic embedding, and (2) integrating aesthetic conditions into the denoising process through value-mixed cross-attention, with the network connected by zero-initialized linear layers. Our key insight is to enhance the aesthetic presentation of existing diffusion models by designing a superior condition control method, all while preserving the image-text alignment. Through our meticulous design, VMix is flexible enough to be applied to community models for better visual performance without retraining. To validate the effectiveness of our method, we conducted extensive experiments, showing that VMix outperforms other state-of-the-art methods and is compatible with other community modules (e.g., LoRA, ControlNet, and IPAdapter) for image generation. The project page is https://vmix-diffusion.github.io/VMix/.

文生图扩散模型美学增强交叉注意力

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。