通过代理评估提升视觉语言模型评分标准质量,实现更可靠的奖励建模。
Rationale Matters: Learning Transferable Rubrics via Proxy-Guided Critique for VLM Reward Models
- 用轻量级代理模型评估评分标准的合理性,形成可优化的反馈信号。
- 仅用5万样本即达到顶尖性能,优于四倍数据训练的方法。
- 学到的评分标准能迁移到新评估者,测试时无需额外训练。
用于视觉语言模型(VLM)的生成式奖励模型(GRMs)通常采用三阶段流程:生成评分标准、基于标准打分、给出最终判断。但中间的评分标准很少被直接优化。以往工作要么将其视为附属产物,要么依赖昂贵且不可微的LLM人工评判,缺乏有效训练信号。本文提出Proxy-GRM,引入代理引导的评分标准验证机制,结合强化学习显式提升评分标准质量。具体地,训练轻量级代理(Proxy-SFT和Proxy-RL),输入候选评分标准、原始查询及偏好对,仅凭评分标准预测偏好顺序。代理预测准确率作为评分标准质量的奖励信号,激励模型生成内部一致且可迁移的评分标准。在约5万样本下,Proxy-GRM在VL-Reward Bench、Multimodal Reward Bench和MM-RLHF-Reward Bench上均达当前最优,超越使用四倍数据训练的方法。消融实验表明,Proxy-SFT比Proxy-RL更具判别力,隐式奖励聚合效果最佳。关键优势在于,学习到的评分标准可迁移至未见评估者,在测试阶段无需再训练即可提升奖励准确性。代码已开源。
原文摘要 · Abstract (English)
Generative reward models (GRMs) for vision-language models (VLMs) often evaluate outputs via a three-stage pipeline: rubric generation, criterion-based scoring, and a final verdict. However, the intermediate rubric is rarely optimized directly. Prior work typically either treats rubrics as incidental or relies on expensive LLM-as-judge checks that provide no differentiable signal and limited training-time guidance. We propose Proxy-GRM, which introduces proxy-guided rubric verification into Reinforcement Learning (RL) to explicitly enhance rubric quality. Concretely, we train lightweight proxy agents (Proxy-SFT and Proxy-RL) that take a candidate rubric together with the original query and preference pair, and then predict the preference ordering using only the rubric as evidence. The proxy's prediction accuracy serves as a rubric-quality reward, incentivizing the model to produce rubrics that are internally consistent and transferable. With ~50k data samples, Proxy-GRM reaches state-of-the-art results on the VL-Reward Bench, Multimodal Reward Bench, and MM-RLHF-Reward Bench, outperforming the methods trained on four times the data. Ablations show Proxy-SFT is a stronger verifier than Proxy-RL, and implicit reward aggregation performs best. Crucially, the learned rubrics transfer to unseen evaluators, improving reward accuracy at test time without additional training. Our code is available at https://github.com/Qwen-Applications/Proxy-GRM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。