通过融合广度与深度推理，提升生成式评分模型的评估可靠性。
Beyond Length Scaling: Synergizing Breadth and Depth for Generative Reward Models
- 将推理过程拆分为广度与深度两种模式，结构化重构原始理由
- 在五个基准上超越现有开源模型平均8.2%，广度适合主观偏好，深度适合客观正确
- 强化学习机制让模型自动适配任务类型，实现推理风格自适应
生成式评分模型（GRMs）近年来证明，扩展思维链（CoT）长度能显著提升评估可靠性。然而，现有方法多依赖无结构的长度扩展，忽视了不同推理机制的差异性：广度推理（B-CoT，即多维度原则覆盖）与深度推理（D-CoT，即判断严谨性）效能各异。为此，本文提出Mix-GRM框架，通过模块化合成流程将原始理由重构为结构化的B-CoT和D-CoT，并分别采用监督微调（SFT）与可验证奖励的强化学习（RLVR）进行内化与优化。大量实验表明，Mix-GRM在五个基准上达到新最优，平均领先主流开源模型8.2%。结果揭示：B-CoT更适用于主观偏好任务，而D-CoT在客观正确性任务中表现更优；若推理机制与任务不匹配，性能将显著下降。此外，我们发现RLVR具有开关放大效应，促使模型自发分配推理风格以适应任务需求。合成数据与模型已发布于Hugging Face，代码开源至Github。
原文摘要 · Abstract (English)
Recent advancements in Generative Reward Models (GRMs) have demonstrated that scaling the length of Chain-of-Thought (CoT) reasoning considerably enhances the reliability of evaluation. However, current works predominantly rely on unstructured length scaling, ignoring the divergent efficacy of different reasoning mechanisms: Breadth-CoT (B-CoT, i.e., multi-dimensional principle coverage) and Depth-CoT (D-CoT, i.e., substantive judgment soundness). To address this, we introduce Mix-GRM, a framework that reconfigures raw rationales into structured B-CoT and D-CoT through a modular synthesis pipeline, subsequently employing Supervised Fine-Tuning (SFT) and Reinforcement Learning with Verifiable Rewards (RLVR) to internalize and optimize these mechanisms. Comprehensive experiments demonstrate that Mix-GRM establishes a new state-of-the-art across five benchmarks, surpassing leading open-source RMs by an average of 8.2\%. Our results reveal a clear divergence in reasoning: B-CoT benefits subjective preference tasks, whereas D-CoT excels in objective correctness tasks. Consequently, misaligning the reasoning mechanism with the task directly degrades performance. Furthermore, we demonstrate that RLVR acts as a switching amplifier, inducing an emergent polarization where the model spontaneously allocates its reasoning style to match task demands. The synthesized data and models are released at \href{https://huggingface.co/collections/DonJoey/mix-grm}{Hugging Face}, and the code is released at \href{https://github.com/Don-Joey/Mix-GRM}{Github}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。