用少量样本生成奖励模型，无需人工标注偏好数据。
Fake it till You Make it: Reward Modeling as Discriminative Prediction
- 通过对比目标样本与生成输出，实现无监督奖励建模。
- 仅需数百个目标样本即可训练，显著降低数据成本。
- 适用于文本生成、图像生成等场景，适合追求高效训练的研究者。
有效的奖励模型在强化学习后训练中对视觉生成模型的性能提升至关重要。然而，现有方法依赖大量人工标注的偏好数据或繁琐设计的质量维度，实施复杂且不完整。受生成对抗网络（GAN）对抗训练启发，本文提出 GAN-RM 框架，无需人工标注偏好数据或显式质量维度工程。该方法通过区分少量代表性未配对的目标样本（称为偏好代理数据）与模型生成的普通输出来训练奖励模型，仅需几百个目标样本。大量实验表明，GAN-RM 在多个关键应用中均表现优异，包括测试时缩放（采用 Best-of-N 样本筛选）、后训练方法如监督微调（SFT）和直接偏好优化（DPO）。代码与数据将发布于 https://github.com/Visualignment/GAN-RM。
原文摘要 · Abstract (English)
An effective reward model plays a pivotal role in reinforcement learning for post-training enhancement of visual generative models. However, current approaches of reward modeling suffer from implementation complexity due to their reliance on extensive human-annotated preference data or meticulously engineered quality dimensions that are often incomplete and engineering-intensive. Inspired by adversarial training in generative adversarial networks (GANs), this paper proposes GAN-RM, an efficient reward modeling framework that eliminates manual preference annotation and explicit quality dimension engineering. Our method trains the reward model through discrimination between a small set of representative, unpaired target samples(denoted as Preference Proxy Data) and model-generated ordinary outputs, requiring only a few hundred target samples. Comprehensive experiments demonstrate our GAN-RM's effectiveness across multiple key applications including test-time scaling implemented as Best-of-N sample filtering, post-training approaches like Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO). Code and data will be released at https://github.com/Visualignment/GAN-RM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。