提出可扩展的生成奖励建模框架,解决视觉生成中奖励信号失准问题。
RewardDance: Reward Scaling in Visual Generation
- 将奖励建模转化为模型预测'是'令牌的概率,与视觉语言模型架构天然对齐
- 支持高达260亿参数的奖励模型扩展,且在多任务中显著提升生成质量
- 有效防止奖励作弊,缓解模式崩溃,适合大规模生成系统优化
奖励模型(RMs)在强化学习提升生成模型方面至关重要,但其在视觉生成中的规模化机制仍缺乏探索。现有方法受限于CLIP类模型的结构与模态约束,以及布拉德利-特里损失与视觉语言模型(VLM)下一词预测机制的根本错位,阻碍了有效扩展。更严重的是,RLHF优化过程普遍存在奖励作弊问题,即模型利用奖励信号漏洞而未提升真实质量。为此,我们提出RewardDance,一种通过新型生成式奖励范式实现可扩展的奖励建模框架。将奖励分数重构为模型预测“是”令牌的概率,表示生成图像在特定标准下优于参考图像,从而实现与VLM架构的内在对齐。该对齐支持双维扩展:(1) 模型扩展:将奖励模型规模系统性扩大至260亿参数;(2) 上下文扩展:集成任务指令、参考示例与链式思维(CoT)推理。大量实验表明,RewardDance在文本到图像、文本到视频及图像到视频生成任务中显著超越现有最佳方法。关键突破在于解决了长期存在的奖励作弊问题:大规模奖励模型在强化学习微调过程中保持高奖励方差,证明其抗作弊能力,能持续生成多样且高质量输出,显著缓解小模型常见的模式崩溃问题。
原文摘要 · Abstract (English)
Reward Models (RMs) are critical for improving generation models via Reinforcement Learning (RL), yet the RM scaling paradigm in visual generation remains largely unexplored. It primarily due to fundamental limitations in existing approaches: CLIP-based RMs suffer from architectural and input modality constraints, while prevalent Bradley-Terry losses are fundamentally misaligned with the next-token prediction mechanism of Vision-Language Models (VLMs), hindering effective scaling. More critically, the RLHF optimization process is plagued by Reward Hacking issue, where models exploit flaws in the reward signal without improving true quality. To address these challenges, we introduce RewardDance, a scalable reward modeling framework that overcomes these barriers through a novel generative reward paradigm. By reformulating the reward score as the model's probability of predicting a "yes" token, indicating that the generated image outperforms a reference image according to specific criteria, RewardDance intrinsically aligns reward objectives with VLM architectures. This alignment unlocks scaling across two dimensions: (1) Model Scaling: Systematic scaling of RMs up to 26 billion parameters; (2) Context Scaling: Integration of task-specific instructions, reference examples, and chain-of-thought (CoT) reasoning. Extensive experiments demonstrate that RewardDance significantly surpasses state-of-the-art methods in text-to-image, text-to-video, and image-to-video generation. Crucially, we resolve the persistent challenge of "reward hacking": Our large-scale RMs exhibit and maintain high reward variance during RL fine-tuning, proving their resistance to hacking and ability to produce diverse, high-quality outputs. It greatly relieves the mode collapse problem that plagues smaller models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。