arXiv:2604.10966cs.CVcs.AI2026-04

一次推理同时评估多个回复,速度提升最多10倍。

You Only Judge Once: Multi-response Reward Modeling in a Single Forward Pass

  • 将多个回复拼接后统一评分,实现高效多路比较。
  • 在6个基准上达到顶尖性能,视频任务速度提升10倍。
  • 适合需要快速生成优质回复的场景,如对话系统优化。

我们提出一种判别式多模态奖励模型,可在单次前向传播中对多个候选回复进行评分。传统判别式奖励模型需对每个回复独立评估,每次需一次前向传播。我们的方法将多个回复用分隔符拼接,通过其标量得分计算交叉熵,实现直接的多路偏好学习,并带来最高达$N imes$的墙时速提升和浮点运算量减少。为支持超过现有成对基准的$N$路评估,我们构建了两个新基准:(1) MR$^2$Bench-Image 包含来自8个不同模型的人工标注排序;(2) MR$^2$Bench-Video 是基于94K条众包成对人类判断的大型视频问答奖励基准,经偏好图集成去噪。两个基准均提供从完整排序中采样的4回复评估变体。模型基于40亿参数的视觉-语言骨干网络,采用LoRA微调与轻量MLP值头,在六个多模态奖励基准(包括MR$^2$Bench-Image、MR$^2$Bench-Video及其他四个基准)上表现领先,优于现有更大规模的生成式与判别式奖励模型。进一步实验表明,该奖励模型在强化学习中结合GRPO使用,可生成更稳定、质量更高的策略模型,显著优于单回复判别式奖励模型基线。

原文摘要 · Abstract (English)

We present a discriminative multimodal reward model that scores all candidate responses in a single forward pass. Conventional discriminative reward models evaluate each response independently, requiring multiple forward passes, one for each potential response. Our approach concatenates multiple responses with separator tokens and applies cross-entropy over their scalar scores, enabling direct comparative reasoning and efficient $N$-way preference learning. The multi-response design also yields up to $N\times$ wall-clock speedup and FLOPs reduction over conventional single-response scoring. To enable $N$-way reward evaluation beyond existing pairwise benchmarks, we construct two new benchmarks: (1) MR$^2$Bench-Image contains human-annotated rankings over responses from 8 diverse models; (2) MR$^2$Bench-Video is a large-scale video-based reward benchmark derived from 94K crowdsourced pairwise human judgments over video question-answering spanning 19 models, denoised via preference graph ensemble. Both benchmarks provide 4-response evaluation variants sampled from the full rankings. Built on a 4B vision-language backbone with LoRA fine-tuning and a lightweight MLP value head, our model achieves state-of-the-art results on six multimodal reward benchmarks, including MR$^2$Bench-Image, MR$^2$Bench-Video, and four other existing benchmarks. Our model outperforms existing larger generative and discriminative reward models. We further demonstrate that our reward model, when used in reinforcement learning with GRPO, produces improved policy models that maintain performance across standard multimodal benchmarks while substantially improving open-ended generation quality, outperforming a single-response discriminative reward model (RM) baseline by a large margin in both training stability and open-ended generation quality.

奖励建模多回复评分效率优化视觉语言

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。