解决图文生成中的条件冲突问题,提升生成准确率。
BideDPO: Conditional Image Generation with Simultaneous Text and Condition Alignment
- 分步解耦文本与条件信号,减少训练时梯度干扰。
- 在双对齐基准上实现文本成功率提升35%,条件遵循度显著提高。
- 适合需要精确控制图像结构与风格的生成任务研究者。
条件图像生成通过引入结构、空间或风格先验增强文本到图像合成,但现有方法难以处理两类冲突:1)输入级冲突(条件图与文本提示矛盾),2)模型偏差冲突(生成偏见破坏对齐,即使条件与文本一致)。标准监督微调难以应对这些复杂情况。基于偏好优化的DPO虽有潜力,却受限于文本与条件信号的梯度纠缠,且缺乏多约束任务的解耦数据。为此,我们提出双向解耦DPO框架(BideDPO),构建两组独立的偏好对(分别针对条件和文本),降低梯度耦合。采用自适应损失平衡策略调控两组影响,实现均衡优化。设计自动化数据流水线,采样模型输出并生成冲突感知数据,嵌入迭代优化流程中持续精炼模型与数据。构建DualAlign基准评估文本与条件间的冲突解决能力。实验表明,BideDPO显著提升文本成功率达+35%以上,同时强化条件遵循。在COCO数据集上也验证了有效性。
原文摘要 · Abstract (English)
Conditional image generation enhances text-to-image synthesis with structural, spatial, or stylistic priors, but current methods face challenges in handling conflicts between sources. These include 1) input-level conflicts, where the conditioning image contradicts the text prompt, and 2) model-bias conflicts, where generative biases disrupt alignment even when conditions match the text. Addressing these conflicts requires nuanced solutions, which standard supervised fine-tuning struggles to provide. Preference-based optimization techniques like Direct Preference Optimization (DPO) show promise but are limited by gradient entanglement between text and condition signals and lack disentangled training data for multi-constraint tasks. To overcome this, we propose a bidirectionally decoupled DPO framework (BideDPO). Our method creates two disentangled preference pairs-one for the condition and one for the text-to reduce gradient entanglement. The influence of pairs is managed using an Adaptive Loss Balancing strategy for balanced optimization. We introduce an automated data pipeline to sample model outputs and generate conflict-aware data. This process is embedded in an iterative optimization strategy that refines both the model and the data. We construct a DualAlign benchmark to evaluate conflict resolution between text and condition. Experiments show BideDPO significantly improves text success rates (e.g., +35%) and condition adherence. We also validate our approach using the COCO dataset. Project Pages: https://limuloo.github.io/BideDPO/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。