用卡尔曼滤波优化语言模型推理中的奖励估计，提升强化学习效果。
Kalman Filter Enhanced GRPO for Reinforcement Learning-Based Language Model Reasoning
- 将每组奖励视为噪声观测，用一维卡尔曼滤波估计隐藏的基准奖励和不确定性。
- 在数学推理任务上，相比GRPO，KRPO显著提升训练收益曲线和最终准确率。
- 无需新增参数，计算开销小，适合部署在资源受限的推理场景中。
优势函数是强化学习中降低策略梯度估计方差的核心概念。对于语言建模，分组相对策略优化（GRPO）提出以组内样本均值作为优势归一化的基线。该估计器对小组规模小和回溯级随机性敏感，可能导致某些场景下优势估计次优。本文提出卡尔曼滤波增强的分组相对策略优化（KRPO），一种轻量级变体，将每组奖励视为对潜在提示级奖励基线的噪声观测，并使用一维卡尔曼滤波同时估计基线及其不确定性。KRPO不引入额外可学习参数，可低开销集成至GRPO。在数学推理基准测试中，KRPO始终优于GRPO，提升了训练奖励曲线与最终准确率。结果表明，自适应优势估计是无评判器强化学习在语言模型推理中的有前景方向。代码已公开于 https://github.com/billhhh/KRPO_LLMs_RL。
原文摘要 · Abstract (English)
The advantage function is a central concept in RL that helps reduce variance in policy gradient estimates. For language modeling, Group Relative Policy Optimization (GRPO) was proposed to use the within-group sample mean as a baseline for advantage normalization. This estimator can be sensitive to small group size and rollout-level stochasticity, which may lead to suboptimal advantage estimates in some settings. In this paper, we propose Kalman Filter Enhanced Group Relative Policy Optimization (KRPO), a lightweight variant that treats per-group rewards as noisy observations of a latent prompt-level reward baseline and uses a 1D Kalman filter to estimate both the baseline and its uncertainty. KRPO introduces no additional learned parameters and can be integrated into GRPO with minimal computational overhead. On mathematical reasoning benchmarks, KRPO consistently improves training reward curves and final accuracy over GRPO. These results suggest that adaptive advantage estimation is a promising direction for critic-free reinforcement learning in language model reasoning. The code is available at https://github.com/billhhh/KRPO_LLMs_RL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。