让大模型主动学习知识，显著提升记忆准确率。
Learning Facts at Scale with Active Reading
- 设计自生成学习策略的主动阅读框架，提升知识吸收能力。
- 在SimpleQA上达到66%准确率，比普通微调高313%。
- 适用于专家领域知识训练，适合需要高精度事实推理的场景。
大语言模型虽能存储海量知识，但其知识学习与回忆往往不可靠，依赖训练数据中事实的出现频率等未明因素。当前缺乏有效工具确保模型稳定掌握特定知识。为此，我们提出主动阅读（Active Reading）框架：训练模型基于给定材料自动生成学习策略。实验表明，在专家领域任务中，该方法显著优于普通微调及其他数据增强手段。我们训练的80亿参数专家模型在Wikipedia基准的SimpleQA子集上达到66%准确率（较普通微调提升313%相对），在FinanceBench上达26%（相对提升160%）。此外，主动阅读可扩展至预训练阶段，我们发布了Meta WikiExpert-8B——一个基于1万亿生成令牌训练的维基百科专家模型，在事实问答任务上超越数百亿参数的模型。
原文摘要 · Abstract (English)
LLMs are known to store vast amounts of knowledge in their parametric memory. However, learning and recalling facts from this memory is known to be unreliable, depending largely on the prevalence of particular facts in the training data and other factors which are poorly understood. Practitioners are lacking tools which will allow them to ensure that the models learn a given body of knowledge reliably and consistently. To this end, we propose Active Reading: a framework where we train models to study a given set of material with self-generated learning strategies. First, we demonstrate models trained with Active Reading on expert domains absorb significantly more knowledge than vanilla finetuning and other data augmentations. We train expert 8B models that achieve 66% on a Wikipedia-grounded subset of SimpleQA (+313% relative over vanilla finetuning) and 26% on FinanceBench (+160% relative over vanilla finetuning) by applying Active Reading to the source documents for each benchmark. Finally, we show that Active Reading can be utilized at pre-training scale to build more factual models. As a demonstration of this, we release Meta WikiExpert-8B, a Wikipedia-expert model trained on 1 trillion generated tokens, which outcompetes models with hundreds of billions of parameters on factual QA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。