让扩散模型生成的图片风格更一致,支持更优的DiT架构。
AlignedGen: Aligning Style Across Generated Images
- 提出ShiftPE解决DiT中位置信号冲突问题
- 通过注意力共享提升多图风格一致性
- 可灵活使用外部图像作风格参考,适合创意设计
尽管生成能力强大,扩散模型在相同风格提示下生成的图像仍难以保持风格一致,限制了其在创作流程中的应用。现有无需训练的方法受限于U-Net架构,导致结果质量低、出现物体重复等伪影,且不兼容更优的Diffusion Transformer(DiT)。为此,我们提出AlignedGen,一种全新的无需训练框架,可提升DiT模型生成图像间的风格一致性。研究首次揭示:直接共享注意力在DiT中因位置嵌入不当引发冲突而失效。为此提出移位位置嵌入(ShiftPE),通过为每张图分配非重叠的位置索引化解冲突。在此基础上,设计先进注意力共享(AAS)三类技术,充分释放注意力共享潜力。此外,提出高效查询、键、值特征提取算法,使方法能无缝引入外部图像作为风格参考。大量实验验证,该方法显著提升生成图像间风格一致性,同时保持精准的文本-图像对齐。
原文摘要 · Abstract (English)
Despite their generative power, diffusion models struggle to maintain style consistency across images conditioned on the same style prompt, hindering their practical deployment in creative workflows. While several training-free methods attempt to solve this, they are constrained to the U-Net architecture, which not only leads to low-quality results and artifacts like object repetition but also renders them incompatible with superior Diffusion Transformer (DiT). To address these issues, we introduce AlignedGen, a novel training-free framework that enhances style consistency across images generated by DiT models. Our work first reveals a critical insight: naive attention sharing fails in DiT due to conflicting positional signals from improper position embeddings. We introduce Shifted Position Embedding (ShiftPE), an effective solution that resolves this conflict by allocating a non-overlapping set of positional indices to each image. Building on this foundation, we develop Advanced Attention Sharing (AAS), a suite of three techniques meticulously designed to fully unleash the potential of attention sharing within the DiT. Furthermore, to broaden the applicability of our method, we present an efficient query, key, and value feature extraction algorithm, enabling our method to seamlessly incorporate external images as style references. Extensive experimental results validate that our method effectively enhances style consistency across generated images while maintaining precise text-to-image alignment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。