提升大模型强化学习中的状态价值估计精度，让训练更稳定高效。
Hista and Numca: Estimate State Value Effectively for LLM Reinforcement Learning
- 用数值跨度做可度量的阶段标记，精准定位状态价值
- 通过隐藏状态加权平均多轮回放结果，提升估值准确性
- 适用于各类大模型和强化学习算法，计算开销小
强化学习通过奖励信号直接优化大语言模型行为，但状态价值估计在大模型后训练中仍缺乏研究。本文构建了状态价值估计基准（SVEB），发现标准方法如PPO中的评判器会退化为粗粒度的组平均基线。为此，提出两种新方法：Numca利用数值跨度作为可度量的里程碑进行状态价值估计；Hista则基于大模型隐藏状态，对不相交的回放轨迹及其回报进行加权平均。大量实验表明，两种方法均显著提升状态价值估计精度，并在不同强化学习算法与模型规模下增强训练表现，且计算开销几乎无增加。
原文摘要 · Abstract (English)
Reinforcement learning (RL) refines large language models (LLMs) by directly optimizing model behavior through reward signals. While accurate state value estimation is critical for stable training in classical RL, it remains an underexplored challenge in LLM post-training. In this work, we introduce the State Value Estimation Benchmark (SVEB) to assess state estimation within existing RL frameworks and show that critics in standard approaches like PPO collapse to a coarse group-average baseline. To address this, we propose two techniques: Numca, which leverages numerical spans as gradable milestones for state value estimation, and Hista, a framework that uses LLM's hidden states as representation to weighted average disjoint rollouts and their return. Extensive experiments demonstrate that both methods yield more accurate state value estimates and enhance training performance across different RL algorithms and model sizes without incurring significant computational overhead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。