arXiv:2606.07500cs.LGcs.AI2026-06

通过稀疏专家分解实现任务无关持续学习,缓解遗忘问题。

Sparse Subspace-to-Expert Sharing for Task-Agnostic Continual Learning

论文配图:Sparse Subspace-to-Expert Sharing for Task-Agnostic Continual Learning
图 1 · 摘自论文原文
  • 将参数分为任务专属专家和共享专家,动态隔离知识
  • 在LLaMA-2 7B和Qwen3-4B上保持早期任务知识并提升反向迁移
  • 适配不同领域基准,支持自动路由检索专家组合

大型语言模型的持续学习受限于塑性-稳定性困境:习得新能力常导致旧知识灾难性遗忘。现有方法对参数处理均质化,无法区分特定任务知识与共享能力。本文提出面向任务无关持续学习的稀疏专家混合框架SETA,通过自适应稀疏子空间分解构建任务专属专家模块。与传统更新中任务争夺同一参数不同,SETA将知识分离为专属专家(捕捉任务特异性模式)与共享专家(提取共性特征)。该结构通过自适应弹性锚定与路由感知正则化共同维护权重与路由层面的共享知识,并由统一门控网络在推理时自动调用正确专家组合。在多个领域专用基准上的实验表明,SETA在性能上达到或优于当前最优持续学习基线,尤其在保留早期任务知识方面表现优异,并在LLaMA-2 7B和Qwen3-4B上显著提升反向迁移效果。

原文摘要 · Abstract (English)

Continual learning in Large Language Models (LLMs) is hindered by the plasticity-stability dilemma, where acquiring new capabilities often leads to catastrophic forgetting of previous knowledge. Existing methods typically treat parameters uniformly, failing to distinguish between specific task knowledge and shared capabilities. We introduce Mixture of Sparse Experts for Task Agnostic Continual Learning (SETA), a framework that resolves the plasticity-stability conflict through adaptive sparse subspace decomposition into task-specific expert modules. Unlike standard updates, where tasks compete for the same parameters, SETA separates knowledge into unique experts, designed to isolate task-specific patterns, and shared experts, responsible for capturing common features. This structure is maintained through adaptive elastic anchoring and a routing-aware regularization that jointly protect shared knowledge at both the weight and routing levels and enable a unified gating network to automatically retrieve the correct expert combination during inference. Extensive experiments across diverse domain-specific benchmarks demonstrate that SETA achieves competitive or superior overall performance relative to state-of-the-art continual learning baselines, with particularly strong retention of early-task knowledge and improved backward transfer on LLaMA-2 7B and Qwen3-4B.

持续学习专家混合大模型知识保留

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。