用强化学习让大模型真正理解新知识，而非死记硬背。
From Memorization to Absorption: Mixed-Policy RL for Continual Knowledge Injection
- 采用混合策略强化学习，通过黄金答案提供学习信号。
- 在复杂推理和多源检索任务上显著优于监督微调方法。
- 适合需要持续更新知识的智能系统研发人员参考。
持续知识注入对保持大语言模型在快速变化世界中的时效性至关重要。现有方法依赖监督微调（SFT），虽能记忆注入事实，但无法泛化至改写、文档组合和推理任务。为此，我们提出黄金GRPO注入（GRIN），一种三阶段自学习框架。黄金GRPO是一种专为知识注入设计的混合策略强化学习算法，在策略回放失败于新事实时仍能通过黄金答案提供学习信号。我们还引入了空白（Blank）和反事实（Counter）两个文档级基准，分别评估新知识获取与反事实覆盖能力，涵盖单事实回忆、多源检索和推理性推理。实验表明：混合策略强化学习可实现超越监督微调的知识吸收效果。GRIN在更难的问题类型上显著优于SFT和混合策略强化学习基线，同时在基础事实回忆上表现相当。
原文摘要 · Abstract (English)
Continual knowledge injection is essential for keeping large language models up-to-date in a fast-evolving world. Existing methods rely on supervised fine-tuning (SFT), which memorizes injected facts in their training format but fails to generalize across paraphrasing, document combinations, and reasoning. To address this, we propose Golden-GRPO Injection (GRIN), a three-stage self-learning framework for continual knowledge injection. Golden-GRPO is a mixed-policy reinforcement learning algorithm designed specifically for knowledge injection, which injects a golden answer to provide learning signal even when on-policy rollouts fail on novel facts. We further introduce Blank and Counter, two document-level benchmarks targeting novel acquisition and counterfactual overwrite respectively, each evaluating single-fact recall, multi-source retrieval, and inferential reasoning. Our experiments establish a clear empirical claim: mixed-policy reinforcement learning enables knowledge absorption beyond what supervised fine-tuning can achieve. GRIN substantially outperforms SFT and mixed-policy RL baselines on the harder question types while matching them on basic fact recall.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。