arXiv:2410.11817cs.CVcs.LG2024-10ICLR被引 21

解决长文本生成图像时对齐难题,提升图像与长文本匹配度。

Improving Long-Text Alignment for Text-to-Image Diffusion Models

  • 将长文本分段编码,突破预训练模型输入长度限制。
  • 分解偏好评分机制,有效降低微调过拟合问题。
  • 在20小时微调后超越更强基座模型的图文对齐效果。

文本到图像扩散模型虽已取得惊人成果,但面对更长的文本输入时,现有编码方法如CLIP面临瓶颈,导致图像与长文本对齐困难。为此,我们提出LongAlign,包含分段编码和分解偏好优化两种方法。分段编码将长文本分割为多个片段分别处理,突破预训练编码器的最大输入长度限制。在偏好优化方面,我们分析基于CLIP的偏好模型得分机制,发现其可分解为文本相关部分(衡量图文对齐)与文本无关部分(评估其他视觉偏好)。进一步发现,文本无关部分在微调中引发普遍过拟合问题。为此,我们提出重加权策略,为两部分分配不同权重,有效缓解过拟合并提升对齐性能。使用本方法对512×512 Stable Diffusion v1.5进行约20小时微调后,生成结果在图文对齐上优于更强的基座模型PixArt-α和Kandinsky v2.2。代码已开源。

原文摘要 · Abstract (English)

The rapid advancement of text-to-image (T2I) diffusion models has enabled them to generate unprecedented results from given texts. However, as text inputs become longer, existing encoding methods like CLIP face limitations, and aligning the generated images with long texts becomes challenging. To tackle these issues, we propose LongAlign, which includes a segment-level encoding method for processing long texts and a decomposed preference optimization method for effective alignment training. For segment-level encoding, long texts are divided into multiple segments and processed separately. This method overcomes the maximum input length limits of pretrained encoding models. For preference optimization, we provide decomposed CLIP-based preference models to fine-tune diffusion models. Specifically, to utilize CLIP-based preference models for T2I alignment, we delve into their scoring mechanisms and find that the preference scores can be decomposed into two components: a text-relevant part that measures T2I alignment and a text-irrelevant part that assesses other visual aspects of human preference. Additionally, we find that the text-irrelevant part contributes to a common overfitting problem during fine-tuning. To address this, we propose a reweighting strategy that assigns different weights to these two components, thereby reducing overfitting and enhancing alignment. After fine-tuning $512 \times 512$ Stable Diffusion (SD) v1.5 for about 20 hours using our method, the fine-tuned SD outperforms stronger foundation models in T2I alignment, such as PixArt-$α$ and Kandinsky v2.2. The code is available at https://github.com/luping-liu/LongAlign.

文本生成扩散模型图文对齐

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。