用测试时计算重用预训练数据，显著提升模型性能。
Reusing Pre-Training Data at Test Time is a Compute Multiplier
- 测试时通过检索增强生成，激活预训练数据中未被充分挖掘的知识。
- 在MMLU上效果提升约5倍于仅靠预训练的计算增益。
- 适合关注模型效率与知识利用的研究者和工程师。
大语言模型从庞大的预训练语料中学习，具备解决多样化任务的能力；然而，尽管研究者不断改进数据集，却很少评估预训练过程对数据信息的挖掘效率。本文通过检索增强生成结合测试时计算，量化了预训练过程中遗留的数据价值及其随规模的变化。结果表明，预训练后从标准且广泛开源的数据集中检索，可在MMLU、Math-500和SimpleQA上带来显著准确率提升，且在去污染后仍有效。在MMLU上，检索相当于约5倍的计算增益。进一步利用测试时额外算力解析检索内容，使公开的LLaMA 3.1 8B模型在MMLU上提升10个百分点。整体结果表明，当前预训练方法未能充分利用现有数据集中的信息，仍有巨大优化空间。
原文摘要 · Abstract (English)
Large language models learn from their vast pre-training corpora, gaining the ability to solve an ever increasing variety of tasks; yet although researchers work to improve these datasets, there is little effort to understand how efficient the pre-training apparatus is at extracting ideas and knowledge from the data. In this work, we use retrieval augmented generation along with test-time compute as a way to quantify how much dataset value was left behind by the process of pre-training, and how this changes across scale. We demonstrate that pre-training then retrieving from standard and largely open-sourced datasets results in significant accuracy gains in MMLU, Math-500, and SimpleQA, which persist through decontamination. For MMLU we observe that retrieval acts as a ~5x compute multiplier versus pre-training alone. We show that these results can be further improved by leveraging additional compute at test time to parse the retrieved context, demonstrating a 10 percentage point improvement on MMLU for the public LLaMA 3.1 8B model. Overall, our results suggest that today's pre-training methods do not make full use of the information in existing pre-training datasets, leaving significant room for progress.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。