让语言模型用向量查询知识库，生成更准且可追溯的事实。
Co-LMLM: Continuous-Query Limited Memory Language Models
- 用连续向量作为查询键，从知识库中动态获取信息。
- 360M模型在低数据量下表现超越大模型，事实准确率媲美GPT-4o-mini。
- 支持自由文本标注，适用于非维基百科的任意语料训练。
受限记忆语言模型（LMLM）在预训练阶段将事实知识外化至知识库（KB），而非存储于模型权重中。生成时，模型按需从KB中检索知识。该范式赋予知识可控性等优势。本文提出连续查询型LMLM（CO-LMLM），其KB以连续向量键关联文本知识值，突破了以往依赖关系型KB与离散查询的限制。CO-LMLM以极低成本生成灵活向量查询，并将可读、可溯源的知识融入生成过程。我们设计了标注流水线，对任意文本中的自由形式事实片段进行标记，摆脱了先前仅限维基百科的约束。在维基百科和FineWeb-Edu上预训练，涵盖多个模型规模，CO-LMLM在困惑度和事实精确度上均优于以往LMLM及常规LLM。360M规模模型在更低数据量下实现更低困惑度，且在SimpleQA验证下的性能达到gpt-4o-mini水平，高于Claude Sonnet 4.5。
原文摘要 · Abstract (English)
Limited memory language models (LMLMs) externalize factual knowledge during pretraining to a knowledge base (KB), rather than memorizing it in their weights. During generation, the model then fetches knowledge from the KB as needed. This recently introduced paradigm provides multiple advantages, including knowledge control capabilities that remain beyond conventional LLMs. We propose continuous-query LMLM (CO-LMLM), where the KB pairs continuous keys with textual knowledge values, a significant departure from prior reliance on relational KB and queries. CO-LMLM generates flexible vector queries at minimal cost, while still integrating human-readable and attributable retrieved knowledge into its generation. We pair this design with an annotation pipeline that tags free-form factual spans in arbitrary text, removing prior work's restriction to Wikipedia. Across pretraining on Wikipedia and FineWeb-Edu and at multiple model scales, CO-LMLM outperforms prior LMLMs and vanilla LLMs in both perplexity and factual precision. At 360M scale, this includes lower perplexity than models pretrained on 40x more data, and SimpleQA-verified performance that is in line with gpt-4o-mini and higher than Claude Sonnet 4.5.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。