提出新基准与训练方法，提升大模型对更新知识的记住与推理能力。
Memorization vs. Reasoning: Updating LLMs with New Knowledge
- 用自动生成的'记忆令牌'引导训练，让模型更好记住新知识。
- 在复杂推理任务中，新方法使记忆准确率提升25.4%。
- 适合关注知识更新与推理能力的大模型研究者。
大型语言模型（LLMs）在其参数中编码了大量预训练知识，但随着现实信息演变，如何更新它们仍是挑战。现有方法和评估基准主要针对实体替换，难以覆盖复杂的现实动态。本文提出知识更新模拟平台KUP，自动构建反映真实知识演化的证据语料库，并设计直接与间接探针，评估任意更新方法在知识记忆与推理上的表现。随后提出轻量级方法记忆条件训练（MCT），在训练时将更新语料中的词元关联到自生成的“记忆”词元。该策略促使模型在推理时主动调用并推理新记忆内容。在两个强模型上的实验表明：(1) KUP基准极具挑战性，最优持续预训练（CPT）模型在间接探针任务中表现不足2%；(2) MCT显著优于先前的持续预训练基线，在直接探针任务中记忆准确率最高提升25.4%。
原文摘要 · Abstract (English)
Large language models (LLMs) encode vast amounts of pre-trained knowledge in their parameters, but updating them as real-world information evolves remains a challenge. Existing methodologies and benchmarks primarily target entity substitutions, failing to capture the full breadth of complex real-world dynamics. In this paper, we introduce Knowledge Update Playground (KUP), an automatic pipeline for simulating realistic knowledge updates reflected in an evidence corpora. KUP's evaluation framework includes direct and indirect probes to both test memorization of updated facts and reasoning over them, for any update learning methods. Next, we present a lightweight method called memory conditioned training (MCT), which conditions tokens in the update corpus on self-generated "memory" tokens during training. Our strategy encourages LLMs to surface and reason over newly memorized knowledge at inference. Our results on two strong LLMs show that (1) KUP benchmark is highly challenging, with the best CPT models achieving $<2\%$ in indirect probing setting (reasoning) and (2) MCT training significantly outperforms prior continued pre-training (CPT) baselines, improving direct probing (memorization) results by up to $25.4\%$.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。