通过动态调温增强跨模态注意力，提升文本图像生成对齐精度
Rethinking Cross-Modal Interaction in Multimodal Diffusion Transformers
- 引入温度调节机制动态平衡文本与图像令牌的注意力
- 在T2I-CompBench上显著改善物体外观、属性绑定和空间关系对齐
- 无需大量参数，适配FLUX、SD3.5等主流扩散模型
多模态扩散变换器（MM-DiT）在文本驱动图像生成中取得显著进展，但即使最先进的模型如FLUX仍难以实现文本提示与生成内容间的精准对齐。我们发现MM-DiT注意力机制存在两大问题：1）视觉与文本模态间令牌不平衡导致跨模态注意力被抑制；2）缺乏时间步感知的注意力加权，阻碍对齐。为此，我们提出 extbf{温度调节跨模态注意力（TACA）}，一种参数高效的方法，通过温度缩放和时间步依赖调整动态重平衡多模态交互。结合LoRA微调，TACA在T2I-CompBench基准上显著提升文本-图像对齐，计算开销极小。我们在FLUX和SD3.5等先进模型上验证了其有效性，可改进物体外观、属性绑定及空间关系的对齐。结果表明，平衡跨模态注意力对提升文图生成语义保真度至关重要。代码已开源。
原文摘要 · Abstract (English)
Multimodal Diffusion Transformers (MM-DiTs) have achieved remarkable progress in text-driven visual generation. However, even state-of-the-art MM-DiT models like FLUX struggle with achieving precise alignment between text prompts and generated content. We identify two key issues in the attention mechanism of MM-DiT, namely 1) the suppression of cross-modal attention due to token imbalance between visual and textual modalities and 2) the lack of timestep-aware attention weighting, which hinder the alignment. To address these issues, we propose \textbf{Temperature-Adjusted Cross-modal Attention (TACA)}, a parameter-efficient method that dynamically rebalances multimodal interactions through temperature scaling and timestep-dependent adjustment. When combined with LoRA fine-tuning, TACA significantly enhances text-image alignment on the T2I-CompBench benchmark with minimal computational overhead. We tested TACA on state-of-the-art models like FLUX and SD3.5, demonstrating its ability to improve image-text alignment in terms of object appearance, attribute binding, and spatial relationships. Our findings highlight the importance of balancing cross-modal attention in improving semantic fidelity in text-to-image diffusion models. Our codes are publicly available at \href{https://github.com/Vchitect/TACA}
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。