强化学习能提升大模型对隐含知识的直接记忆能力。
Beyond Reasoning: Reinforcement Learning Unlocks Parametric Knowledge in LLMs
- 用奖励信号优化参数知识分布，不依赖推理链
- 平均提升27%准确率，优于训练和推理优化方法
- 特别擅长唤醒极难样本中的隐藏答案
强化学习（RL）在大模型推理中表现卓越，但其是否能提升对参数化知识的直接回忆仍不清楚。我们在一个受控的零样本、单跳、闭卷问答设置中研究该问题，不使用思维链，仅以二元正确性奖励进行训练，并通过事实级训练测试去重，确保性能提升来自回忆而非推理或记忆。在三个模型家族和多个事实问答基准上，RL带来约27%的平均相对增益，超越了训练和推理时的基线方法。机制分析显示，RL主要重新分配已有知识的概率质量，而非获取新事实，将正确答案从低概率尾部转移到可靠贪婪生成位置。数据归因研究表明，最难样本最具信息量：那些在128次预强化采样中从未出现的答案（仅占约18%的训练数据）贡献了约83%的性能提升，因为这些罕见正确轨迹在训练中仍会生成并被强化。综合来看，这些发现拓展了强化学习的作用，将其重新定位为解锁而非获取潜在参数知识的工具。
原文摘要 · Abstract (English)
Reinforcement learning (RL) has achieved remarkable success in LLM reasoning, but whether it can also improve direct recall of parametric knowledge remains an open question. We study this question in a controlled zero-shot, one-hop, closed-book QA setting with no chain-of-thought, training only on binary correctness rewards and applying fact-level train-test deduplication to ensure gains reflect improved recall rather than reasoning or memorization. Across three model families and multiple factual QA benchmarks, RL yields ~27% average relative gains, surpassing both training- and inference-time baselines alike. Mechanistically, RL primarily redistributes probability mass over existing knowledge rather than acquiring new facts, moving correct answers from the low-probability tail into reliable greedy generations. Our data-attribution study reveals that the hardest examples are the most informative: those whose answers never appear in 128 pre-RL samples (only ~18% of training data) drive ~83% of the gain, since rare correct rollouts still emerge during training and get reinforced. Together, these findings broaden the role of RL beyond reasoning, repositioning it as a tool for unlocking rather than acquiring latent parametric knowledge.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。