用偏好学习提升文本生成图像的组合准确性
Compositional Text-to-Image Generation Via Region-aware Bimodal Direct Preference Optimization
- 构建大规模高质量偏好数据集BiComp，指导模型理解复杂文本指令
- 联合优化图像与文本偏好，显著提升对组合概念的生成精度
- 区域级引导聚焦关键视觉区域，适合需要精准布局的任务
尽管文本到图像（T2I）模型进展迅速，但准确生成包含属性绑定、对象关系和计数等复杂组合提示的图像仍具挑战。为此，我们提出BiDPO框架，以增强T2I模型在组合文本生成方面的能力。首先，设计了一套精心构建的流水线，创建了大规模、严格质量控制的偏好数据集BiComp；随后，将扩散强化学习中的直接偏好优化（Diffusion DPO）扩展为联合优化图像与文本偏好的方法，显著提升了模型遵循复杂文本提示的能力。为进一步实现细粒度对齐，引入区域级引导策略，聚焦于与组合概念相关联的视觉区域。实验结果表明，BiDPO在多个基准上显著提升了组合保真度，持续优于现有方法。该方法凸显了基于偏好的微调在复杂文本到图像任务中的潜力，提供了一种灵活且可扩展的替代方案。
原文摘要 · Abstract (English)
Despite the rapid progress of text-to-image (T2I) models, generating images that accurately reflect complex compositional prompts (covering attribute bindings, object relationships, counting) still remains challenging. To address this, we propose BiDPO, a framework to enhance T2I model's capability of compositional text-to-image generation. We begin by introducing an carefully designed pipeline to construct a large-scale preference dataset, BiComp, with strictly quality control. Then, we extend Diffusion DPO to jointly optimize image and text preferences, which is shown to greatly effective in improving the models to follow complex text prompt in generation. To further enhance the models for fine-grained alignment, we employ a region-level guidance method to focus on regions relevant to compositional concepts. Experimental results demonstrate that our BiDPO substantially improves compositional fidelity, consistently outperforming prior methods across multiple benchmarks. Our approach highlights the potential of preference-based fine-tuning for complex text-to-image tasks, offering a flexible and scalable alternative to existing techniques.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。