提出CRD框架,让扩散模型更精准地理解文本指令并减少奖励漏洞。
Diffusion Reinforcement Learning via Centered Reward Distillation
- 基于前向过程的奖励蒸馏,通过中心化消除归一化常数干扰。
- 在文本到图像任务中实现快速收敛与低奖励滥用,优于当前SOTA。
- 适合需要高精度文本对齐和稳定训练的生成模型微调场景。
扩散与流模型在生成任务上表现卓越,但对细粒度提示一致性、组合正确性及文本渲染等行为的建模仍依赖于评分或流匹配预训练目标,难以精准控制。尽管强化学习微调可借助外部奖励修复,但扩散强化学习常不稳定:基于轨迹的方法内存开销大且梯度方差高;前向过程方法收敛快却易出现分布漂移和奖励劫持。本文提出中心化奖励蒸馏(CRD),基于KL正则化的奖励最大化构建前向过程微调框架。关键洞察在于,提示内中心化使不可计算的归一化常数相消,从而得到良好定义的奖励匹配目标。为实现可靠的文本到图像微调,引入三项技术:(i)解耦采样器与移动参考以防止比例信号崩溃;(ii)KL锚定至CFG引导的预训练模型,控制长期漂移并保持推理语义一致;(iii)自适应奖励强度的KL权重,在大正则化下加速早期学习,同时减少后期对奖励模型漏洞的利用。在 exttt{GenEval}和 exttt{OCR}奖励下的文本到图像后训练实验表明,CRD在未见偏好指标上实现了具有竞争力的SOTA奖励优化效果,收敛迅速且奖励劫持显著降低。
原文摘要 · Abstract (English)
Diffusion and flow models achieve State-Of-The-Art (SOTA) generative performance, yet many practically important behaviors such as fine-grained prompt fidelity, compositional correctness, and text rendering are weakly specified by score or flow matching pretraining objectives. Reinforcement Learning (RL) fine-tuning with external, black-box rewards is a natural remedy, but diffusion RL is often brittle. Trajectory-based methods incur high memory cost and high-variance gradient estimates; forward-process approaches converge faster but can suffer from distribution drift, and hence reward hacking. In this work, we present \textbf{Centered Reward Distillation (CRD)}, a diffusion RL framework derived from KL-regularized reward maximization built on forward-process-based fine-tuning. The key insight is that the intractable normalizing constant cancels under \emph{within-prompt centering}, yielding a well-posed reward-matching objective. To enable reliable text-to-image fine-tuning, we introduce techniques that explicitly control distribution drift: (\textit{i}) decoupling the sampler from the moving reference to prevent ratio-signal collapse, (\textit{ii}) KL anchoring to a CFG-guided pretrained model to control long-run drift and align with the inference-time semantics of the pre-trained model, and (\textit{iii}) reward-adaptive KL strength to accelerate early learning under large KL regularization while reducing late-stage exploitation of reward-model loopholes. Experiments on text-to-image post-training with \texttt{GenEval} and \texttt{OCR} rewards show that CRD achieves competitive SOTA reward optimization results with fast convergence and reduced reward hacking, as validated on unseen preference metrics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。