让扩散模型在推理时自由调节生成偏好，无需重训。
ParetoSlider: Diffusion Models Post-Training for Continuous Reward Control
- 用连续变化的权重训练单个模型，逼近所有最优权衡点。
- 在三个主流模型上表现优于固定权重基线，且支持精细控制。
- 适合需要灵活调整生成目标的图像编辑场景。
强化学习后训练已成为对齐生成模型与人类偏好的标准方法，但多数方法依赖单一标量奖励。当多个标准需兼顾时，现有方法通过‘早期标量化’将奖励合并为固定加权和，导致模型在训练时即锁定单一权衡点，无法在推理时调整本质冲突的目标（如提示遵循度与源图像保真度）。我们提出ParetoSlider，一种多目标强化学习框架，使单个扩散模型能逼近完整的帕累托前沿。通过将连续变化的偏好权重作为条件信号进行训练，用户可在推理时自由导航最优权衡，无需重训或维护多个检查点。我们在三种先进流匹配骨干模型（SD3.5、FluxKontext、LTX-2）上评估该方法，结果表明：单一偏好条件模型的表现达到或超过针对固定奖励权衡单独训练的基线，同时唯一实现了对竞争性生成目标的细粒度控制。
原文摘要 · Abstract (English)
Reinforcement Learning (RL) post-training has become the standard for aligning generative models with human preferences, yet most methods rely on a single scalar reward. When multiple criteria matter, the prevailing practice of ``early scalarization'' collapses rewards into a fixed weighted sum. This commits the model to a single trade-off point at training time, providing no inference-time control over inherently conflicting goals -- such as prompt adherence versus source fidelity in image editing. We introduce ParetoSlider, a multi-objective RL (MORL) framework that trains a single diffusion model to approximate the entire Pareto front. By training the model with continuously varying preference weights as a conditioning signal, we enable users to navigate optimal trade-offs at inference time without retraining or maintaining multiple checkpoints. We evaluate ParetoSlider across three state-of-the-art flow-matching backbones: SD3.5, FluxKontext, and LTX-2. Our single preference-conditioned model matches or exceeds the performance of baselines trained separately for fixed reward trade-offs, while uniquely providing fine-grained control over competing generative goals.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。