研究预训练与检索的权衡，发现小模型靠检索提效果，大模型更依赖自身知识。
To Memorize or to Retrieve: Scaling the Interaction Between Pretraining and Retrieval
- 通过控制模型大小和检索数据源，系统分析预训练与检索的交互机制。
- 91%的性能提升在每参数一个检索词时达成，且大模型在准确率上收益更大。
- 检索从已见过数据中获取信息仍有效，提示应区分内部记忆与外部访问。
检索增强生成（RAG）通过在测试时提供相关上下文来提升语言模型在知识密集型任务中的表现。本文系统研究了预训练与检索之间的权衡关系，基于OLMo-2训练了从30M到3B参数的语言模型，覆盖高达100B DCLM tokens的预训练数据，同时调整预训练数据规模、检索库大小及检索来源（预训练数据或新数据），在推理、科学问答和开放域问答基准上进行评估。结果表明，检索增益依赖于模型容量和预训练暴露程度，且呈强前加载特征：每模型参数仅需一个检索词即可实现最大提升的91%。但这种互动具有任务依赖性——小模型在黄金答案困惑度上提升更明显，而大模型、高预训练模型在准确率上获益更多。从已有数据中检索仍保留大部分增益。因此，检索是任务、训练范式和评估指标相关的补充手段，其价值取决于数据存储规模和信息新颖性。总体而言，这支持在语言模型设计中明确划分数据的内化与外部访问路径。
原文摘要 · Abstract (English)
Retrieval-augmented generation (RAG) improves language model (LM) performance by providing relevant context at test time for knowledge-intensive situations. In this work, we systematically study the trade-off between pretraining and retrieval by training OLMo-2-based LMs ranging from 30M to 3B parameters on up to 100B DCLM tokens, while varying pretraining data scale, retrieval store size, and retrieval store source (pretraining vs. new data) across reasoning, scientific QA, and open-domain QA benchmarks. We find that retrieval gains depend on model capacity and pretraining exposure and are strongly front-loaded, with a median 91% of the largest observed improvement realized by one retrieval token per model parameter. However, the interaction is objective-dependent: smaller models gain more in gold-answer perplexity, whereas larger, more-pretrained models gain more in accuracy. Retrieval from previously seen data also preserves most of the held-out retrieval gain. Retrieval is therefore a task-, regime-, and metric-dependent complement to parametric learning whose value also depends on datastore size and information novelty. Overall, this motivates the explicit partitioning of data between internalization and external access for LM design.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。