让扩散模型同时优化多个审美维度，避免偏好冲突。
Multi-dimensional Preference Alignment by Conditioning Reward Itself
- 用条件化奖励机制解耦不同评价维度的优化方向。
- 在Stable Diffusion 1.5和SDXL上显著提升多维对齐效果。
- 推理时可动态控制特定维度，无需额外训练。
基于人类反馈的强化学习已成为对齐扩散模型的标准方法。然而，我们发现标准DPO公式存在根本缺陷：其依赖布拉德利-特雷西模型将美学质量、语义一致性等多样评估维度聚合为单一标量奖励，导致奖励冲突——当某些特征在全局非优选样本中出现时，模型被迫遗忘这些有益属性。为此，我们提出多奖励条件DPO（MCDPO），通过引入解耦的布拉德利-特雷西目标，将偏好结果向量作为训练条件注入，使模型能在单个网络中独立学习各奖励维度的优化方向。我们进一步设计维度奖励丢弃机制，确保各维度均衡优化。在Stable Diffusion 1.5和SDXL上的大量实验表明，MCDPO在基准测试中表现更优。特别地，该条件框架结合无分类器引导，在推理阶段实现对特定奖励维度的动态、多轴控制，无需额外训练或外部奖励模型。
原文摘要 · Abstract (English)
Reinforcement Learning from Human Feedback has emerged as a standard for aligning diffusion models. However, we identify a fundamental limitation in the standard DPO formulation because it relies on the Bradley-Terry model to aggregate diverse evaluation axes like aesthetic quality and semantic alignment into a single scalar reward. This aggregation creates a reward conflict where the model is forced to unlearn desirable features of a specific dimension if they appear in a globally non-preferred sample. To address this issue, we propose Multi Reward Conditional DPO (MCDPO). This method resolves reward conflicts by introducing a disentangled Bradley-Terry objective. MCDPO explicitly injects a preference outcome vector as a condition during training, which allows the model to learn the correct optimization direction for each reward axis independently within a single network. We further introduce dimensional reward dropout to ensure balanced optimization across dimensions. Extensive experiments on Stable Diffusion 1.5 and SDXL demonstrate that MCDPO achieves superior performance on benchmarks. Notably, our conditional framework enables dynamic and multiple-axis control at inference time using Classifier Free Guidance to amplify specific reward dimensions without additional training or external reward models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。