提升统一图像生成模型对多指令的遵循能力，无需训练即可增强指令准确性。
Scale Your Instructions: Enhance the Instruction-Following Fidelity of Unified Image Generation Model by Self-Adaptive Attention Scaling
- 通过自适应注意力缩放动态调整每个子指令的关注度。
- 在多指令任务中显著提升指令遵循准确率，优于现有方法。
- 适用于需要精准理解复杂指令的图像生成与编辑场景。
近期统一图像生成模型（如OmniGen）可在单一框架内处理多种图像生成与编辑任务，支持自由形式的多模态文本与图像输入。该架构无需文本编码器，大幅降低模型复杂度并统一任务标准，更易使用。然而，我们发现其存在文本指令忽视问题，尤其在包含多个子指令时更为明显。通过扰动分析定位关键步骤与层，并观察交叉注意力图，发现被忽略的子指令与输入图像激活之间存在显著冲突。为此，提出自适应注意力缩放（SaaS）方法，利用相邻时间步间交叉注意力的一致性，动态调节各子指令的注意力激活。SaaS无需额外训练或测试时优化，即可显著提升指令遵循保真度。在基于指令的图像编辑与视觉条件图像生成任务上验证有效，性能优于现有方法。代码已开源：https://github.com/zhouchao-ops/SaaS。
原文摘要 · Abstract (English)
Recent advancements in unified image generation models, such as OmniGen, have enabled the handling of diverse image generation and editing tasks within a single framework, accepting multimodal, interleaved texts and images in free form. This unified architecture eliminates the need for text encoders, greatly reducing model complexity and standardizing various image generation and editing tasks, making it more user-friendly. However, we found that it suffers from text instruction neglect, especially when the text instruction contains multiple sub-instructions. To explore this issue, we performed a perturbation analysis on the input to identify critical steps and layers. By examining the cross-attention maps of these key steps, we observed significant conflicts between neglected sub-instructions and the activations of the input image. In response, we propose Self-Adaptive Attention Scaling (SaaS), a method that leverages the consistency of cross-attention between adjacent timesteps to dynamically scale the attention activation for each sub-instruction. Our SaaS enhances instruction-following fidelity without requiring additional training or test-time optimization. Experimental results on instruction-based image editing and visual conditional image generation validate the effectiveness of our SaaS, showing superior instruction-following fidelity over existing methods. The code is available https://github.com/zhouchao-ops/SaaS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。