动态选择数学推理中最有用的知识点,提升强化学习效果。
AdaKP: Online Adaptive Knowledge-Point Selection for Reasoning-Oriented Reinforcement Learning
- 在线动态筛选最有效的知识提示,而非固定注入。
- 用熵减少量快速评估知识点价值,比传统方法快且准确。
- 适合做竞赛级数学推理的模型优化,无需改动训练器。
基于可验证奖励的强化学习是激发大语言模型推理能力的有效范式,但在竞赛级数学题上仍面临严重奖励稀疏问题。现有方法通常一次性注入原子知识点(KPs)——从标准解法中提炼出的短语提示,但或固定选择,或仅增加文本总量,忽略了最关键的选择维度:在何时选择哪些知识点。本文提出AdaKP,一种在强化学习训练过程中持续动态重选知识点子集的在线选择器。其核心是一个熵代理指标,通过单次前向传播计算知识点带来的下一个词熵减,具备截断偏差的理论保证,远快于依赖采样的估计。为使该信号可用于在线更新,设计了三项轻量机制:动量平滑器抑制每步噪声、退役-复苏管理器剔除弱知识点同时保留探索性、自适应调度器将重评估集中在训练初期。此外,引入预检验证门,在启动昂贵运行前用留一法真值检验代理指标,将方法风险转化为可验证的检查。实现为标准DAPO+GRPO训练器的全加性扩展,无需修改优化器。在全部8个竞赛数学基准上,相比强基线静态选择方法均有提升,且开销极低,证明了在线、可验证的知识点子集选择是推理导向强化学习中一个实用且尚未充分探索的方向。
原文摘要 · Abstract (English)
Reinforcement learning with verifiable rewards is a powerful paradigm for eliciting reasoning in large language models, yet it suffers from severe reward sparsity on competition-level mathematics. A common remedy injects atomic knowledge points (KPs) - short natural-language hints distilled from gold solutions - into the prompt. Existing methods, however, either fix this selection once offline or merely scale the monolithic quantity of injected text, leaving untouched the most informative axis of choice: which subset of atomic KPs to inject, and when. We introduce AdaKP, an online selector that re-chooses each problem's KP subset over the course of RL training. At its core is an entropy proxy that scores a KP by the reduction in next-token entropy it induces - a single inexpensive forward pass, with a provable bound on its truncation bias - in place of expensive rollout-based estimation. Three lightweight mechanisms make this signal usable online: a momentum smoother that absorbs per-step noise, a retirement-and-revival manager that prunes weak KPs while preserving exploration, and an adaptive scheduler that front-loads re-evaluations into early training. AdaKP further contributes a pre-flight validation gate that certifies the proxy against a leave-one-out ground truth before any expensive run is launched, turning method-level risk into a falsifiable check. Realized as a fully additive fork of a standard DAPO+GRPO trainer with no optimizer changes, AdaKP improves over a strong static-selection baseline on all eight competition-mathematics benchmarks at negligible added cost, positioning online, validated KP-subset selection as a practical and as-yet under-explored axis for reasoning-oriented reinforcement learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。