无需额外数据，用大模型修正图像生成与提示的错位。
IMG: Calibrating Diffusion Models via Implicit Multimodal Guidance
- 用多模态大模型识别图文错位，再通过隐式对齐器调整生成过程。
- 在SDXL、SDXL-DPO和FLUX上均超越现有方法，提升对齐精度。
- 可作为插件增强已有微调方法，适配研究者快速优化生成效果。
确保扩散模型生成的图像与输入提示之间的精确多模态对齐长期面临挑战。早期方法依赖高质量偏好数据微调扩散权重，但数据稀缺且难以扩展；近期基于编辑的方法虽能细化局部区域，却可能损害整体图像质量。本文提出隐式多模态引导（IMG），一种无需额外数据或编辑操作的重生成式对齐框架。给定生成图像及其提示，IMG首先利用多模态大语言模型（MLLM）识别错位；其次引入隐式对齐器，操控扩散条件特征以减少错位并支持重生成；最后将对齐目标形式化为可训练的迭代更新偏好目标（Iteratively Updated Preference Objective）。在SDXL、SDXL-DPO和FLUX上的大量定性与定量评估表明，IMG优于现有对齐方法。此外，IMG可作为灵活的即插即用适配器，无缝增强已有微调型对齐方法。代码将于https://github.com/SHI-Labs/IMG-Multimodal-Diffusion-Alignment发布。
原文摘要 · Abstract (English)
Ensuring precise multimodal alignment between diffusion-generated images and input prompts has been a long-standing challenge. Earlier works finetune diffusion weight using high-quality preference data, which tends to be limited and difficult to scale up. Recent editing-based methods further refine local regions of generated images but may compromise overall image quality. In this work, we propose Implicit Multimodal Guidance (IMG), a novel re-generation-based multimodal alignment framework that requires no extra data or editing operations. Specifically, given a generated image and its prompt, IMG a) utilizes a multimodal large language model (MLLM) to identify misalignments; b) introduces an Implicit Aligner that manipulates diffusion conditioning features to reduce misalignments and enable re-generation; and c) formulates the re-alignment goal into a trainable objective, namely Iteratively Updated Preference Objective. Extensive qualitative and quantitative evaluations on SDXL, SDXL-DPO, and FLUX show that IMG outperforms existing alignment methods. Furthermore, IMG acts as a flexible plug-and-play adapter, seamlessly enhancing prior finetuning-based alignment methods. Our code will be available at https://github.com/SHI-Labs/IMG-Multimodal-Diffusion-Alignment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。