对比多种大模型知识编辑方法，发现基于上下文的推理更可靠。
Benchmarking and Rethinking Knowledge Editing for Large Language Models
- 用上下文推理实现简单有效的知识更新，不依赖参数修改。
- 在真实推理场景下，传统方法表现差，新方法在多任务上均胜出。
- 涵盖事件、通用任务等复杂数据集，评估更贴近实际应用。
知识编辑旨在更新大语言模型中嵌入的知识。然而，现有方法（无论是参数修改还是外部记忆整合）常因评估目标和实验设置不一致而难以比较。为此，本文开展全面基准测试，引入事件级数据集和来自其他任务的通用数据集，覆盖指令微调与推理导向的LLM，在真实的自回归推理设置下评估，而非教师强制解码。除单次编辑外，还评估多编辑场景以反映实际需求。采用四个评估维度，包括可迁移性，并将所有近期方法与一个简单基线模型——选择性上下文推理（SCR）进行对比。实证结果表明，基于参数的编辑方法在真实条件下表现不佳；而SCR在所有设置中持续优于其他方法。本研究揭示了当前知识编辑方法的局限性，凸显基于上下文推理作为更鲁棒替代方案的潜力。
原文摘要 · Abstract (English)
Knowledge editing aims to update the embedded knowledge within Large Language Models (LLMs). However, existing approaches, whether through parameter modification or external memory integration, often suffer from inconsistent evaluation objectives and experimental setups. To address this gap, we conduct a comprehensive benchmarking study. In addition to fact-level datasets, we introduce more complex event-based datasets and general-purpose datasets drawn from other tasks. Our evaluation covers both instruction-tuned and reasoning-oriented LLMs, under a realistic autoregressive inference setting rather than teacher-forced decoding. Beyond single-edit assessments, we also evaluate multi-edit scenarios to better reflect practical demands. We employ four evaluation dimensions, including portability, and compare all recent methods against a simple and straightforward baseline named Selective Contextual Reasoning (SCR). Empirical results reveal that parameter-based editing methods perform poorly under realistic conditions. In contrast, SCR consistently outperforms them across all settings. This study offers new insights into the limitations of current knowledge editing methods and highlights the potential of context-based reasoning as a more robust alternative.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。