让大模型持续学习不遗忘，通过拆分专家模块实现。
Split-on-Share: Mixture of Sparse Experts for Task-Agnostic Continual Learning
- 将模型分为任务专属专家和共享专家，避免参数竞争。
- 在多个基准上表现优于现有方法，显著减少遗忘。
- 适合需要长期更新但又不想重训的AI应用。
大语言模型在持续学习中面临可塑性与稳定性之间的矛盾：学习新知识常导致旧知识灾难性遗忘。现有方法通常对所有参数一视同仁，无法区分特定任务知识与共享能力。本文提出面向任务无关持续学习的稀疏专家混合框架SETA，通过将模型分解为模块化子空间来解决该矛盾。不同于传统更新方式中任务争夺同一参数，SETA将知识分离为专用专家（捕捉任务特异性模式）和共享专家（提取通用特征）。通过弹性权重锚定机制保护关键共享知识，并由统一门控网络在推理时自动选择正确专家组合。在多样领域特定及通用基准上的大量实验表明，SETA持续优于当前基于参数高效微调的先进持续学习方法。
原文摘要 · Abstract (English)
Continual learning in Large Language Models (LLMs) is hindered by the plasticity-stability dilemma, where acquiring new capabilities often leads to catastrophic forgetting of previous knowledge. Existing methods typically treat parameters uniformly, failing to distinguish between specific task knowledge and shared capabilities. We introduce Mixture of Sparse Experts for Task-Agnostic Continual Learning, referred to as SETA, a framework that resolves the plasticity-stability conflict by decomposing the model into modular subspaces. Unlike standard updates, where tasks compete for the same parameters, SETA separates knowledge into unique experts, designed to isolate task-specific patterns, and shared experts, responsible for capturing common features. This structure is maintained through elastic weight anchoring, which protects critical shared knowledge and enables a unified gating network to automatically retrieve the correct expert combination for each task during inference. Extensive experiments across diverse domain-specific and general benchmarks demonstrate that SETA consistently outperforms state-of-the-art parameter-efficient fine-tuning-based continual learning methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。