通过动态调整文本引导,提升多模态生成模型的图文对齐效果。
TexTailor: Inference-Time Textual Guidance Tailoring for Multimodal Diffusion Transformers
- 在推理时针对不同层增强文本隐状态,优化图文交互。
- 使文本-图像匹配度提升,如SD3.5上T2I-Combench达63.00%。
- 适用于精准编辑与加速生成,兼容多种应用场景。
基于Transformer的扩散模型(如FLUX和Qwen Image)在视觉生成中取得突破,但其仅依赖文本条件与视觉特征的交互生成图像。当这种交互无法体现提示的复杂结构时,生成结果可能不理想。为此,本文系统分析各模块功能,通过移除、禁用或增强特定层的文本隐状态,发现:1)语义信息出现在早期层,细节在后期层生成;2)移除某层通常比禁用文本影响小;3)在特定层增强文本条件可改善语义属性。基于此,提出 extit{method},一种推理时的块级文本引导定制方法,不仅提升图文一致性,还支持精确编辑与推理加速。大量实验表明,该方法优于多个基线,在SD3.5上将T2I-Combench从56.92%提升至63.00%,GenEval从66.42%提升至71.63%,且不牺牲生成质量。
原文摘要 · Abstract (English)
Recent breakthroughs of transformer-based diffusion models, particularly with Multimodal Diffusion Transformers (MMDiT) driven models like FLUX and Qwen Image, have facilitated thrilling experiences in visual generation. However, these models rely only on the interactions between textual conditions and visual features to produce semantically aligned images. Once the interactions fail to reflect the nuanced compositional structure of the prompt, the generated images might be unsatisfactory. Thus, a comprehensive understanding of how different blocks and their interactions with textual conditions is crucial for better understanding the intrinsic attributes and for enhancing their interactions accordingly to strengthen the prompts adherence. In this paper, we first develop a systematic pipeline to comprehensively investigate each block's functionality by \textit{removing}, \textit{disabling}, and \textit{enhancing} textual hidden-states at corresponding blocks. Our analysis reveals that 1) semantic information appears in earlier blocks and finer details are rendered in later blocks, 2) removing specific blocks is usually less disruptive than disabling text conditions, and 3) enhancing textual conditions in selective blocks improves semantic attributes. Building on these observations, we propose \method, a novel inference-time method for tailoring block-wise textual guidance. Our approach not only improves text-image alignment but also enables a range of downstream applications, including precise editing and inference acceleration. Extensive experiments demonstrated that our method outperforms various baselines and remains flexible across text-to-image generation, image editing, and inference acceleration. Our method improves T2I-Combench from 56.92\% to 63.00\% and GenEval from 66.42\% to 71.63\% on SD3.5, without sacrificing synthesis quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。