用最少必要知识提升大模型推理能力,解决奖励稀疏难题
KnowRL: Boosting LLM Reasoning via Reinforcement Learning with Minimal-Sufficient Knowledge Guidance

- 将提示拆解为最小原子知识点,通过约束搜索生成紧凑训练集
- 1.5B模型在8个基准上达74.16%准确率,比基线高9.63点
- 适合追求高效推理的开发者与研究者,代码开源可复现
强化学习虽能提升大模型推理能力,但在复杂问题上常因奖励稀疏而效果受限。现有基于提示的方法通过注入部分解或抽象模板缓解此问题,但通常增加更多标记,带来冗余、不一致和额外训练开销。我们提出知觉强化学习(KnowRL),将提示设计视为最小充分性引导问题。训练中,KnowRL将引导分解为原子知识点(KPs),并使用约束子集搜索(CSS)构建紧凑且交互感知的子集。我们发现剪枝存在悖论——移除一个KPs可能有益,但同时移除多个反而有害,因此显式优化在此依赖结构下的鲁棒子集筛选。我们在OpenMath-Nemotron-1.5B基础上训练KnowRL-Nemotron-1.5B。在1.5B规模下八个推理基准测试中,KnowRL-Nemotron-1.5B持续优于强基线。推理时无KPs提示下平均准确率达70.08%,较原模型提升+9.63点;启用精选KPs后性能升至74.16%,刷新该规模新纪录。模型、训练数据及代码已公开于https://github.com/Hasuer/KnowRL。
原文摘要 · Abstract (English)
RLVR improves reasoning in large language models, but its effectiveness is often limited by severe reward sparsity on hard problems. Recent hint-based RL methods mitigate sparsity by injecting partial solutions or abstract templates, yet they typically scale guidance by adding more tokens, which introduce redundancy, inconsistency, and extra training overhead. We propose \textbf{KnowRL} (Knowledge-Guided Reinforcement Learning), an RL training framework that treats hint design as a minimal-sufficient guidance problem. During RL training, KnowRL decomposes guidance into atomic knowledge points (KPs) and uses Constrained Subset Search (CSS) to construct compact, interaction-aware subsets for training. We further identify a pruning interaction paradox -- removing one KP may help while removing multiple such KPs can hurt -- and explicitly optimize for robust subset curation under this dependency structure. We train KnowRL-Nemotron-1.5B from OpenMath-Nemotron-1.5B. Across eight reasoning benchmarks at the 1.5B scale, KnowRL-Nemotron-1.5B consistently outperforms strong RL and hinting baselines. Without KP hints at inference, KnowRL-Nemotron-1.5B reaches 70.08 average accuracy, already surpassing Nemotron-1.5B by +9.63 points; with selected KPs, performance improves to 74.16, establishing a new state of the art at this scale. The model, curated training data, and code are publicly available at https://github.com/Hasuer/KnowRL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。