让AI画画时更听懂复杂描述，动态调整文本与图像的互动
Diff-Aid: Inference-time Adaptive Interaction Denoising for Rectified Text-to-Image Generation
- 推理时动态调节每层和每步的文本-图像交互强度
- 在SD 3.5和FLUX上显著提升描述符合度与画面质量
- 可插拔集成，适合需要精准控制生成的场景
近期文本到图像扩散模型虽取得显著进展，但忠实遵循复杂文本描述仍具挑战，主要源于文本与视觉特征间交互不足。已有方法通过架构设计或手工加权文本条件增强交互，但缺乏灵活性，且忽略不同模块与去噪阶段间的动态交互。为此，我们提出Diff-Aid，一种轻量级推理时自适应方法，可动态调整每个注意力头中文本与图像的交互强度，覆盖所有Transformer块与去噪时间步。该方法不仅提升生成质量，还产生可解释的调制模式，揭示不同模块、时间步与文本词元对语义对齐的贡献。作为即插即用模块，Diff-Aid可无缝集成至下游应用，如风格LoRA、可控生成与零样本编辑。在强基线（SD 3.5和FLUX）上的实验表明，其在提示遵循度、视觉质量与人类偏好等指标上均实现一致提升。代码与模型将公开。
原文摘要 · Abstract (English)
Recent text-to-image (T2I) diffusion models have achieved remarkable advancement, yet faithfully following complex textual descriptions remains challenging due to insufficient interactions between textual and visual features. Prior approaches enhance such interactions via architectural design or handcrafted textual condition weighting, but lack flexibility and overlook the dynamic interactions across different blocks and denoising stages. To provide a more flexible and efficient solution to this problem, we propose Diff-Aid, a lightweight inference-time method that adaptively adjusts per-token text and image interactions across transformer blocks and denoising timesteps. Beyond improving generation quality, Diff-Aid yields interpretable modulation patterns that reveal how different blocks, timesteps, and textual tokens contribute to semantic alignment during denoising. As a plug-and-play module, Diff-Aid can be seamlessly integrated into downstream applications for further improvement, including style LoRAs, controllable generation, and zero-shot editing. Experiments on strong baselines (SD 3.5 and FLUX) demonstrate consistent improvements in prompt adherence, visual quality, and human preference across various metrics. Our code and models will be released.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。