根据模型内部不确定性动态触发思维链,提升推理效率与准确性。
Reason Only When Needed: Efficient Generative Reward Modeling via Model-Internal Uncertainty

- 用并行生成的收敛行为估算不确定性,仅在必要时激活思维链。
- 在多个基准上降低推理成本,同时提升答案准确率。
- 无需人工设计特征,适合需要高效推理的复杂任务场景。
近期生成式奖励模型(GRM)通过思维链(CoT)提示提升了大模型的推理能力。然而现有方法存在两大局限:其一,对所有输入无差别使用CoT,复杂度低的任务也产生额外计算开销;其二,依赖投票机制评估推理路径,评估粒度不足。本文提出E-GRM,一种基于模型内部不确定性的高效生成式奖励建模框架。E-GRM利用并行生成的收敛行为估计不确定性,仅在必要时触发CoT推理,无需人工特征或任务依赖信号。为提升奖励精度,引入轻量级判别评分器,采用混合回归-排序目标训练,实现对推理路径的细粒度评估。在多个推理基准上的实验表明,E-GRM显著降低推理开销,同时持续提升准确率,证明模型内部不确定性是高效、通用的推理感知奖励建模信号。
原文摘要 · Abstract (English)
Recent advancements in the Generative Reward Model (GRM) have demonstrated its potential to enhance the reasoning abilities of LLMs through Chain-of-Thought (CoT) prompting. Despite these gains, existing implementations of GRM suffer from two critical limitations. First, CoT prompting is applied indiscriminately to all inputs regardless of their inherent complexity. This introduces unnecessary computational costs for tasks amenable to fast, direct inference. Second, existing approaches primarily rely on voting-based mechanisms to evaluate CoT outputs, which often lack granularity and precision in assessing reasoning quality. In this paper, we propose E-GRM, an efficient generative reward modeling framework grounded in model-internal uncertainty. E-GRM leverages the convergence behavior of parallel model generations to estimate uncertainty and selectively trigger CoT reasoning only when needed, without relying on handcrafted features or task-dependent signals. To improve reward fidelity, we introduce a lightweight discriminative scorer trained with a hybrid regression--ranking objective to provide fine-grained evaluation of reasoning paths. Experiments on multiple reasoning benchmarks show that E-GRM substantially reduces inference cost while consistently improving answer accuracy, demonstrating that model-internal uncertainty is an effective and general signal for efficient reasoning-aware reward modeling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。