修复MMDiT生成相似主体时的混淆问题,提升图像准确性
Enhancing MMDiT-Based Text-to-Image Models for Similar Subject Generation
- 通过测试时优化,在去噪早期修正模糊潜在表示
- 新数据集上生成成功率超现有方法,质量显著提升
- 适合需要精准生成多个相似对象的研究与应用
作为当前最先进的文生图模型技术,多模态扩散变换器(MMDiT)已显著缓解以往模型的生成缺陷。然而我们发现,当输入提示包含语义或外观相似的多个主体时,该模型仍存在主体忽略或混淆问题。我们识别出MMDiT架构中的三类歧义:块间歧义、文本编码歧义和语义歧义。为此,我们提出在去噪早期通过测试时优化实时修复模糊潜在表示。具体设计了三种损失函数:块对齐损失、文本编码对齐损失和重叠损失,分别对应解决上述三类歧义。尽管改进显著,但生成多个相似主体时语义歧义仍存,因重叠损失引导不足。因此,我们进一步提出重叠在线检测与回溯重采样策略以缓解此问题。在新构建的挑战性相似主体数据集上的实验验证了方法有效性,生成质量更优,成功率达更高。代码将公开于 https://github.com/wtybest/EnMMDiT。
原文摘要 · Abstract (English)
Representing the cutting-edge technique of text-to-image models, the latest Multimodal Diffusion Transformer (MMDiT) largely mitigates many generation issues existing in previous models. However, we discover that it still suffers from subject neglect or mixing when the input text prompt contains multiple subjects of similar semantics or appearance. We identify three possible ambiguities within the MMDiT architecture that cause this problem: Inter-block Ambiguity, Text Encoder Ambiguity, and Semantic Ambiguity. To address these issues, we propose to repair the ambiguous latent on-the-fly by test-time optimization at early denoising steps. In detail, we design three loss functions: Block Alignment Loss, Text Encoder Alignment Loss, and Overlap Loss, each tailored to mitigate these ambiguities. Despite significant improvements, we observe that semantic ambiguity persists when generating multiple similar subjects, as the guidance provided by overlap loss is not explicit enough. Therefore, we further propose Overlap Online Detection and Back-to-Start Sampling Strategy to alleviate the problem. Experimental results on a newly constructed challenging dataset of similar subjects validate the effectiveness of our approach, showing superior generation quality and much higher success rates over existing methods. Our code will be available at https://github.com/wtybest/EnMMDiT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。