将大模型知识拆分到共享核心与可替换记忆模块，提升效率与灵活性。
Knowledge Offloading: Decomposing LLMs into Sparse Backbones and Memory Modules
- 通过结构化稀疏化和轻量恢复模块，分离通用计算与领域知识
- 12%全局稀疏度下性能损失小，远优于无记忆的直接剪枝
- 适合作为高效模型部署、个性化知识更新的方案
大型语言模型将通用能力与特定领域知识统一编码在参数中。本文提出知识卸载（KOFF）框架，将预训练大模型分解为稀疏共享骨干与领域特定记忆模块。基于冻结的基础模型，联合学习结构化裁剪掩码与轻量级恢复模块（采用LoRA适配器和可学习键值缓存）。在3B至8B规模的Llama与Qwen模型上，发现非平凡容量可移出共享骨干而保持模型能力。约12%全局稀疏度下，性能显著优于无记忆的直接剪枝；消融实验表明LoRA与可学习KV记忆互补，专业化分析显示语言特异性神经元优先被移除，而通用神经元多保留在骨干中。结果表明知识可在共享核心与可插拔外部记忆间重新分配。
原文摘要 · Abstract (English)
LLMs encode both general capabilities and domain-specific knowledge in a single set of parameters. We ask whether this capacity can be reorganized: keeping broadly useful computation in a shared backbone, while moving specialized knowledge into external memory modules. We propose \emph{knowledge offloading} (KOFF), a framework for decomposing a pretrained LLM into a sparse shared backbone and domain-specific memories. Starting from a frozen base model, we jointly learn a structured pruning mask and lightweight recovery modules, implemented as LoRA adapters and learned key-value caches. Across Llama and Qwen models from 3B to 8B, we find that non-trivial capacity can be moved out of the shared backbone without a large loss in model ability. At around 12\% global sparsity, KOFF preserves much of the unpruned model's performance, while pruning the same frozen model without memories degrades sharply. Ablations show that LoRA and learned KV memories are complementary, and specialization analyses suggest that the learned decomposition is meaningful: language-specific neurons are preferentially removed while language-general neurons largely remain in the backbone. These results suggest that knowledge can be reallocated between a shared core and swappable external memories.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。