用时序差分优化奖励模型，让大模型强化学习更稳定高效
TDRM: Smooth Reward Models with Temporal Difference for LLM RL and Inference
- 通过最小化时序差分训练更平滑的奖励模型
- 在树搜索中提升23.7%，仅用2.5k数据达基线50.1k效果
- 适合做强化学习和推理验证的大模型开发者
奖励模型在语言模型强化学习与推理时验证中至关重要。但现有模型常缺乏时序一致性，导致策略更新无效、训练不稳定。本文提出TDRM方法，通过最小化时序差分（TD）来训练更平滑可靠的奖励模型，适用于训练时强化学习与推理时验证。实验表明，经TD训练的过程奖励模型（PRMs）在Best-of-N（最高提升6.6%）和树搜索（最高提升23.7%）场景中均表现更优。结合可验证奖励强化学习（RLVR），TD训练的PRMs实现更高数据效率：仅需2.5k数据即可达到基线方法50.1k数据的性能。在8个模型变体（5个系列）中生成更高质量的语言模型策略，包括Qwen2.5-(0.5B, 1.5B)、GLM4-9B-0414、GLM-Z1-9B-0414、Qwen2.5-Math-(1.5B, 7B)和DeepSeek-R1-Distill-Qwen-(1.5B, 7B)。代码已开源：https://github.com/THUDM/TDRM。
原文摘要 · Abstract (English)
Reward models are central to both reinforcement learning (RL) with language models and inference-time verification. However, existing reward models often lack temporal consistency, leading to ineffective policy updates and unstable RL training. We introduce TDRM, a method for learning smoother and more reliable reward models by minimizing temporal differences (TD) for training-time reinforcement learning and inference-time verification. Experiments show that TD-trained process reward models (PRMs) improve performance across Best-of-N (up to 6.6%) and tree-search (up to 23.7%) settings. When combined with Reinforcement Learning with Verifiable Rewards (RLVR), TD-trained PRMs lead to more data-efficient RL -- achieving comparable performance with just 2.5k data to what baseline methods require 50.1k data to attain -- and yield higher-quality language model policies in 8 model variants (5 series), e.g., Qwen2.5-(0.5B, 1,5B), GLM4-9B-0414, GLM-Z1-9B-0414, Qwen2.5-Math-(1.5B, 7B), and DeepSeek-R1-Distill-Qwen-(1.5B, 7B). We release all code at https://github.com/THUDM/TDRM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。