让大模型更会思考如何调用自身知识，提升问答准确率。
Improving Parametric Knowledge Access in Reasoning Language Models
- 用可验证的问答任务作为奖励，训练模型思考如何调用参数中的知识。
- 在TriviaQA上提升9.9%，其他数据集也普遍增益0.6%~4.2%。
- 适合想提升模型知识推理能力的研究者和开发者。
我们研究了语言模型在参数中存储的世界知识的推理访问问题。例如，回忆堪培拉是澳大利亚首都，可通过分析主要城市与专设首都概念来辅助。尽管推理语言模型通过强化学习训练生成数学推理过程，但在调用自身世界知识时表现不佳。我们发现，模型默认不生成最优知识推理；添加简单的“逐步思考”提示即可显著提升知识召回，但对数学无帮助。为此，我们提出使用世界知识问答任务作为可验证奖励，训练模型更好地推理其参数知识。在TriviaQA上经过强化学习后，性能提升9.9%；在Natural Questions、HotpotQA、SimpleQA和StrategyQA上分别提升4.2%、2.1%、0.6%和3.0%。这表明推理模型在参数知识访问方面存在优化不足，但可通过简单训练显著改善。
原文摘要 · Abstract (English)
We study reasoning for accessing world knowledge stored in a language model's parameters. For example, recalling that Canberra is Australia's capital may benefit from thinking through major cities and the concept of purpose-built capitals. While reasoning language models are trained via reinforcement learning to produce reasoning traces on tasks such as mathematics, they may not reason well for accessing their own world knowledge. We first find that models do not generate their best world knowledge reasoning by default: adding a simple "think step-by-step" cue demonstrates statistically significant improvement in knowledge recall but not math. Motivated by this, we propose training models to reason over their parametric knowledge using world-knowledge question answering as a verifiable reward. After reinforcement learning on TriviaQA (+9.9%), performance also improves on Natural Questions, HotpotQA, SimpleQA, and StrategyQA by 4.2%, 2.1%, 0.6%, and 3.0%, respectively. Reasoning models are under-optimized for parametric knowledge access, but can be easily trained to reason better.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。