用分布奖励优化生成模型，提升音视频质量，无需人工标注偏好。
DRAGON: Distributional Rewards Optimize Diffusion Generative Models
- 通过构建实例与分布的对比，实现对生成结果的分布级奖励优化。
- 在20种奖励函数下平均胜率81.45%，人评音乐质量胜率达60.95%。
- 支持跨模态参考（如文本到音频），可灵活设计奖励函数，适合开发者和研究者。
我们提出分布奖励优化框架DRAGON，用于微调媒体生成模型以达成特定目标。相比传统基于人类反馈的强化学习（RLHF）或成对偏好方法（如DPO），DRAGON更具灵活性，可优化个体样本或分布层面的奖励函数，兼容实例级、实例到分布、分布到分布等多种奖励形式。通过选择编码器和一组参考样例构建示例分布，当使用跨模态编码器（如CLAP）时，参考可为不同模态（如文本对音频）。DRAGON收集在线生成样本，用奖励函数评分后构建正负样本集，利用两有限集间的对比实现分布奖励优化。评估中，我们在一个包含20个奖励函数的音频领域文生音乐扩散模型上进行微调，包括自定义音乐美学模型、CLAP得分、Vendi多样性及弗雷歇音频距离（FAD）。同时对比了单曲级与全数据集级FAD设置，并消融多个FAD编码器与参考集。在全部20项目标奖励中，DRAGON平均胜率达81.45%。基于示例集的奖励显著提升生成效果，媲美模型基奖励。在合适示例集下，无需人类偏好标注即可达到60.95%的人类投票质量胜率。该方法为提升人类感知质量提供了新奖励设计范式。
原文摘要 · Abstract (English)
We present Distributional RewArds for Generative OptimizatioN (DRAGON), a versatile framework for fine-tuning media generation models towards a desired outcome. Compared with traditional reinforcement learning with human feedback (RLHF) or pairwise preference approaches such as direct preference optimization (DPO), DRAGON is more flexible. It can optimize reward functions that evaluate either individual examples or distributions of them, making it compatible with a broad spectrum of instance-wise, instance-to-distribution, and distribution-to-distribution rewards. Leveraging this versatility, we construct novel reward functions by selecting an encoder and a set of reference examples to create an exemplar distribution. When cross-modal encoders such as CLAP are used, the reference may be of a different modality (text versus audio). Then, DRAGON gathers online and on-policy generations, scores them with the reward function to construct a positive demonstration set and a negative set, and leverages the contrast between the two finite sets to approximate distributional reward optimization. For evaluation, we fine-tune an audio-domain text-to-music diffusion model with 20 reward functions, including a custom music aesthetics model, CLAP score, Vendi diversity, and Frechet audio distance (FAD). We further compare instance-wise (per-song) and full-dataset FAD settings while ablating multiple FAD encoders and reference sets. Over all 20 target rewards, DRAGON achieves an 81.45% average win rate. Moreover, reward functions based on exemplar sets enhance generations and are comparable to model-based rewards. With an appropriate exemplar set, DRAGON achieves a 60.95% human-voted music quality win rate without training on human preference annotations. DRAGON is a new approach to designing and optimizing reward functions for improving human-perceived quality. Demos at https://ml-dragon.github.io/web
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。