统一了扩散与流模型的奖励微调方法，让优化更简单高效。
Reward Score Matching: Unifying Reward-based Fine-tuning for Flow and Diffusion Models
- 将多种奖励微调法归一为统一的评分匹配框架
- 通过值引导目标实现对齐，减少冗余设计
- 适用于可微与黑盒奖励任务，提升效率与可解释性
基于奖励的微调可使预训练的扩散或流生成模型生成更高奖励的样本，同时保持与预训练模型的接近性。尽管现有方法源于不同视角，我们发现它们均可纳入一个统一流程——奖励评分匹配（RSM）。在此框架下，对齐操作变为对值引导目标的评分匹配，各方法差异主要体现在值引导估计器构建与时间步上的有效优化强度上。该统一视角揭示了现有设计中的偏差-方差-计算权衡，并区分出核心优化组件与无明确收益的辅助机制。基于此，我们在代表性可微与黑盒奖励对齐任务中提出了更简洁高效的重设计。总体而言，RSM将原本零散的奖励微调方法整合为更小、更可理解且更具操作性的设计空间。代码已开源：https://github.com/jaylee2000/rsm。
原文摘要 · Abstract (English)
Reward-based fine-tuning steers a pretrained diffusion or flow-based generative model toward higher-reward samples while remaining close to the pretrained model. Although existing methods are derived from different perspectives, we show that many can be written under a common framework, which we call reward score matching (RSM). Under this view, alignment becomes score matching against a value-guided target, and the main differences across methods reduce to the construction of the value-guidance estimator and the effective optimization strength across timesteps. This unification clarifies the bias-variance-compute tradeoffs of existing designs, and distinguishes core optimization components from auxiliary mechanisms that add complexity without clear benefit. Guided by this perspective, we develop simpler, more efficient redesigns across representative differentiable and black-box reward alignment tasks. Overall, RSM turns a seemingly fragmented collection of reward-based fine-tuning methods into a smaller, more interpretable, and more actionable design space. Code is available at https://github.com/jaylee2000/rsm.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。