arXiv:2503.12575cs.CVcs.AI2025-03被引 7

用投票机制平衡多指标,让图像生成更符合人类偏好。

BalancedDPO: Adaptive Multi-Metric Alignment

  • 通过多评分器投票达成共识,避免单一指标偏差。
  • 在多个数据集上提升偏好胜率,效果优于现有方法。
  • 适合需要多维度优化的图像生成场景。

扩散模型在文本到图像生成中取得显著进展,但其与人类偏好的对齐仍具挑战,因存在语义一致性、美学和人类偏好评分等多重、有时冲突的评估指标。现有对齐方法通常仅优化单一指标或依赖标量奖励聚合,易导致模型偏向特定评价标准。为此,我们提出 BalancedDPO,一种在直接偏好优化(DPO)框架内实现多指标对齐的方案。不同于以往仅依赖单个指标的 DPO 变体,BalancedDPO 引入多个偏好评分器的多数投票共识,并将其直接整合进 DPO 训练循环,配合动态参考模型更新。该共识机制避免了奖励尺度冲突,确保跨异构指标的梯度方向更稳定。在 Pick-a-Pic、PartiPrompt 与 HPD 数据集上的实验表明,BalancedDPO 在 Stable Diffusion 1.5、2.1 和 SDXL 基线模型上均持续提升偏好胜率。全面消融实验进一步验证了多数投票聚合与动态参考更新的优势,凸显该方法在多样化对齐场景中的鲁棒性与泛化能力。

原文摘要 · Abstract (English)

Diffusion models have achieved remarkable progress in text-to-image generation, yet aligning them with human preference remains challenging due to the presence of multiple, sometimes conflicting, evaluation metrics (e.g., semantic consistency, aesthetics, and human preference scores). Existing alignment methods typically optimize for a single metric or rely on scalarized reward aggregation, which can bias the model toward specific evaluation criteria. To address this challenge, we propose BalancedDPO, a framework that achieves multi-metric preference alignment within the Direct Preference Optimization (DPO) paradigm. Unlike prior DPO variants that rely on a single metric, BalancedDPO introduces a majority-vote consensus over multiple preference scorers and integrates it directly into the DPO training loop with dynamic reference model updates. This consensus-based formulation avoids reward-scale conflicts and ensures more stable gradient directions across heterogeneous metrics. Experiments on Pick-a-Pic, PartiPrompt, and HPD datasets demonstrate that BalancedDPO consistently improves preference win rates over the baselines across Stable Diffusion 1.5, Stable Diffusion 2.1 and SDXL backbones. Comprehensive ablations further validate the benefits of majority-vote aggregation and dynamic reference updating, highlighting the method's robustness and generalizability across diverse alignment settings.

扩散模型多指标对齐偏好优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。