arXiv:2605.16686cs.LG2026-05被引 1

提出高效方法,实现稀疏专家模型的知识快速更新。

Scalable Knowledge Editing for Mixture-of-Experts LLMs via Tensor-Structured Updates

  • 基于张量结构与伍德伯里公式,实现专家级精准编辑
  • 编辑速度提升6倍,保持与主流方法相当的准确率
  • 适合需要快速更新知识的大型稀疏模型部署场景

知识编辑(KE)为大语言模型提供了一种轻量级替代方案,避免重复微调。然而,现有方法主要针对密集前馈层,而现代大模型越来越多采用混合专家(MoE)架构以降低内存占用并提升推理效率。这一差异导致大量生产级模型缺乏有效的编辑工具。本文提出一种类似MEMIT的框架,用于MoE模型的知识编辑。该方法利用MoE层的张量结构,在每个专家层面精确建模编辑目标,并应用伍德伯里矩阵恒等式,避免显式构建或求逆完整的专家权重矩阵。所得更新仅需低秩矩阵求逆,无需额外反向传播。实验表明,本方法在主要知识编辑指标上达到强基线水平,同时将编辑速度提升最多6倍,得益于批处理的MEMIT式设计及伍德伯里恒等式带来的低维求逆优势。结果表明,闭式、参数修改型知识编辑可高效扩展至非密集层,为现代稀疏大模型的可扩展知识编辑开辟了道路。

原文摘要 · Abstract (English)

Knowledge editing (KE) provides a lightweight alternative to repeated fine-tuning of LLMs. However, most existing KE methods target dense feed-forward layers, while modern LLMs increasingly adopt Mixture-of-Experts (MoE) architectures for their superior memory footprint and inference efficiency. This mismatch leaves a growing class of production models without principled editing tools. We propose a MEMIT-like framework for knowledge editing in MoE-based LLMs. Our method exploits the tensor structure of MoE layers to formulate the editing objective faithfully at the per expert level, and applies the Woodbury matrix identity to avoid materializing or inverting the full stacked matrix of expert weights. The resulting update reduces to inversions of fixed low-rank matrices and requires no additional backward passes. Empirically, our approach matches the editing quality of strong baselines on the main KE metrics while accelerating the editing procedure by up to 6x, owing to the batched MEMIT-style formulation and the low-dimensional inversions enabled by the Woodbury identity. These results show that closed-form, parameter-modifying KE can be extended efficiently beyond dense layers, opening a path toward scalable knowledge editing in modern sparse LLM architectures.

知识编辑MoE模型高效更新稀疏架构

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。