让语言模型不靠记忆，而是查外部数据库获取知识。
Pre-training Limited Memory Language Models with Internal and External Knowledge
- 训练时屏蔽外部知识，逼模型学会主动查询而非死记硬背。
- 在标准测试上表现接近更大规模的通用大模型。
- 适合需要可编辑、可验证知识的场景，如医疗、法律。
神经语言模型是黑箱——语言模式和事实知识都分散在数十亿个难以理解的参数中。这种纠缠编码使得特定事实的检查、验证或更新变得困难。我们提出有限记忆语言模型（LMLM），一种在预训练阶段将事实知识外置于外部数据库的语言模型，而非在模型权重中记忆。我们的预训练方法通过从训练损失中屏蔽外部检索到的事实值，教导模型进行有针对性的查询，而不是依赖权重中的记忆。实验表明，与显著更大的大型语言模型相比，LMLMs 在标准基准测试上表现相当，同时具备显式、可编辑、可验证的知识库优势。
原文摘要 · Abstract (English)
Neural language models are black-boxes--both linguistic patterns and factual knowledge are distributed across billions of opaque parameters. This entangled encoding makes it difficult to reliably inspect, verify, or update specific facts. We introduce Limited Memory Language Models (LMLM), a new class of language models that externalizes factual knowledge to external database during pre-training rather than memorizing them. Our pre-training approach strategically masks externally retrieved factual values from the training loss, thereby teaching the model to perform targeted lookups rather than relying on memorization in model weights. Our experiments demonstrate that LMLMs achieve competitive performance compared to significantly larger LLMs on standard benchmarks, while offering the advantages of explicit, editable, and verifiable knowledge bases.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。