通过筛选可靠随机种子，提升文本生成图像的组合一致性
All Seeds Are Not Equal: Enhancing Compositional Text-to-Image Generation with Reliable Random Seeds
- 挖掘对组合提示更可靠的初始噪声模式
- 微调后数值与空间组合性能分别提升超19%至60%
- 无需人工标注，适合追求生成稳定性的研究者
文本到图像扩散模型在生成真实图像方面表现出色，但对组合性提示（如“两只狗”或“企鹅在碗的右侧”）常出现结果不一致的问题。本文揭示初始噪声在其中的关键作用：某些噪声模式比其他模式更有利于生成符合组合逻辑的图像。分析显示，不同随机种子会引导模型将物体置于图像中特定区域，可能遵循特定相机角度或构图规律。为提升模型的组合生成能力，我们提出一种方法，自动挖掘这些可靠生成案例，构建无需人工标注的精炼训练集。在该数据集上微调模型后，数值组合任务中，Stable Diffusion 和 PixArt-α 的性能分别提升29.3%和19.5%；空间组合任务中，提升幅度达60.7%和21.1%。
原文摘要 · Abstract (English)
Text-to-image diffusion models have demonstrated remarkable capability in generating realistic images from arbitrary text prompts. However, they often produce inconsistent results for compositional prompts such as "two dogs" or "a penguin on the right of a bowl". Understanding these inconsistencies is crucial for reliable image generation. In this paper, we highlight the significant role of initial noise in these inconsistencies, where certain noise patterns are more reliable for compositional prompts than others. Our analyses reveal that different initial random seeds tend to guide the model to place objects in distinct image areas, potentially adhering to specific patterns of camera angles and image composition associated with the seed. To improve the model's compositional ability, we propose a method for mining these reliable cases, resulting in a curated training set of generated images without requiring any manual annotation. By fine-tuning text-to-image models on these generated images, we significantly enhance their compositional capabilities. For numerical composition, we observe relative increases of 29.3% and 19.5% for Stable Diffusion and PixArt-α, respectively. Spatial composition sees even larger gains, with 60.7% for Stable Diffusion and 21.1% for PixArt-α.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。