arXiv:2602.03584cs.CLcs.AI2026-02被引 4

提出无需更新的通用价值模型,可预判任意模型在新提示下的表现。

$V_0$: A Generalist Value Model for Any Policy at State Zero

  • 用历史指令-性能对动态建模模型能力,作为价值估计的上下文输入
  • 在GRPO训练中提升采样效率,部署时实现性能与成本的帕累托最优路由
  • 适用于大模型训练调度与推理资源分配,尤其适合低成本高效部署

策略梯度方法依赖基线来衡量动作的相对优势,确保模型强化优于当前平均能力的行为。在使用演员-评论家方法(如PPO)训练大语言模型时,该基线通常由与策略模型规模相当的价值模型(评论家)估计。然而,随着策略不断演进,价值模型需昂贵且同步的增量训练以准确追踪策略能力的变化。为避免此开销,组相对策略优化(GRPO)通过一组回滚的平均奖励作为基线,消除了耦合的价值模型;但该方法需要大量采样以维持估计稳定性。本文提出 $V_0$,一个通用价值模型,能够在不进行参数更新的情况下,估计任意模型在未见提示上的预期表现。我们重新定义价值估计,将策略的动态能力显式作为上下文输入,利用历史指令-性能对动态刻画模型状态,突破传统依赖参数拟合感知能力变化的范式。聚焦于初始提示状态(即状态零,故称 $V_0$),该模型作为关键资源调度器:在GRPO训练中,$V_0$ 在回滚前预测成功率,实现高效采样预算分配;在部署阶段,作为路由器,将指令分发至最经济且合适的模型。实验表明,$V_0$ 显著优于启发式预算分配,在大语言模型路由任务中实现了性能与成本的帕累托最优权衡。

原文摘要 · Abstract (English)

Policy gradient methods rely on a baseline to measure the relative advantage of an action, ensuring the model reinforces behaviors that outperform its current average capability. In the training of Large Language Models (LLMs) using Actor-Critic methods (e.g., PPO), this baseline is typically estimated by a Value Model (Critic) often as large as the policy model itself. However, as the policy continuously evolves, the value model requires expensive, synchronous incremental training to accurately track the shifting capabilities of the policy. To avoid this overhead, Group Relative Policy Optimization (GRPO) eliminates the coupled value model by using the average reward of a group of rollouts as the baseline; yet, this approach necessitates extensive sampling to maintain estimation stability. In this paper, we propose $V_0$, a Generalist Value Model capable of estimating the expected performance of any model on unseen prompts without requiring parameter updates. We reframe value estimation by treating the policy's dynamic capability as an explicit context input; specifically, we leverage a history of instruction-performance pairs to dynamically profile the model, departing from the traditional paradigm that relies on parameter fitting to perceive capability shifts. Focusing on value estimation at State Zero (i.e., the initial prompt, hence $V_0$), our model serves as a critical resource scheduler. During GRPO training, $V_0$ predicts success rates prior to rollout, allowing for efficient sampling budget allocation; during deployment, it functions as a router, dispatching instructions to the most cost-effective and suitable model. Empirical results demonstrate that $V_0$ significantly outperforms heuristic budget allocation and achieves a Pareto-optimal trade-off between performance and cost in LLM routing tasks.

大模型价值模型调度推理优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。