arXiv:2607.27610cs.LG2026-07

用卡尔曼滤波动态选提示,让强化学习微调更高效准确。

Kalman Meets Curriculum: Efficient Dynamic Prompt Selection for Adaptive RL Finetuning

论文配图:Kalman Meets Curriculum: Efficient Dynamic Prompt Selection for Adaptive RL Finetuning
图 1 · 摘自论文原文
  • 将提示选择建模为状态估计问题,用卡尔曼滤波追踪难度变化。
  • 在数学、规划等任务上减少83%回滚次数,性能提升0.12点。
  • 无需额外采样,自动适应策略漂移,适合在线微调场景。

强化学习微调显著提升大语言模型的推理能力,但其效果高度依赖于为当前策略选择合适难度的提示。然而,提示难度随训练过程动态变化,现有在线方法面临权衡:基于评估的方法准确但昂贵,基于预测的方法高效但假设难度恒定,不适应RL的非平稳训练动态。为此,我们提出卡尔曼引导提示选择(KGPS),将提示选择重新建模为动态状态估计问题。KGPS在对数空间中使用线性高斯状态空间模型建模每个提示的隐含成功概率,过程噪声与策略更新幅度耦合,使策略大幅变动时不确定性增加。卡尔曼滤波维持提示难度的校准高斯后验分布,通过最大化后验期望训练效用选择提示,偏好中等难度并自然重访不确定提示。该方法自适应策略漂移,且无需额外回滚。在数学、规划和几何推理基准及多种RL算法上的大量实验表明,KGPS在最终准确率和回滚效率上均优于强基线,达到当前在线提示选择方法的最先进水平。例如,在DeepSeek-R1-Distill-7B上,相较于DS方法,KGPS仅使用83%的回滚次数,同时在六个数学推理基准上平均性能提升0.12点。

原文摘要 · Abstract (English)

Reinforcement learning (RL) finetuning significantly enhances the reasoning capabilities of large language models (LLMs), yet its effectiveness critically depends on selecting prompts of appropriate difficulty for the current policy. This is challenging because prompt difficulty evolves throughout training. Existing online methods therefore face a trade-off: evaluation-based approaches are accurate but expensive, while prediction-based approaches are efficient but typically assume stationary difficulty, making them ill-suited to RL's non-stationary training dynamics. To address these issues, we propose a Kalman-Guided Prompt Selection method (KGPS), which reformulates prompt selection as a dynamic state estimation problem rather than static difficulty prediction. KGPS models each prompt's latent success rate in logit space using a linear-Gaussian state-space model, with process noise coupled to the magnitude of policy updates so that uncertainty increases when the policy changes more substantially. A Kalman filter then maintains a calibrated Gaussian posterior over prompt difficulty, and prompts are selected by maximizing a posterior-expected training utility that favors intermediate-difficulty prompts while naturally revisiting uncertain ones. The resulting procedure is adaptive to policy drift and requires no additional rollouts beyond standard policy training. Extensive experiments across mathematics, planning, and geometry reasoning benchmarks, as well as multiple RL algorithms, show that KGPS consistently improves both final accuracy and rollout efficiency over strong baselines, establishing state-of-the-art performance among online prompt selection methods. For example, on DeepSeek-R1-Distill-7B, KGPS uses 83% fewer rollouts than DS while even improving the average performance by 0.12 point across six math reasoning benchmarks.

强化学习提示选择卡尔曼滤波微调优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。