用稀疏专家混合机制让单个提示同时高效处理多任务,避免记忆干扰。
One-Prompt Strikes Back: Sparse Mixture of Experts for Prompt-based Continual Learning
- 将共享提示拆分为多个专家,仅激活相关部分降低干扰。
- 在多个基准上超越专用提示方法,参数量减少超60%。
- 适合资源受限场景的持续学习应用,如边缘设备部署。
基于提示的方法在持续学习中因性能强且内存效率高而受到关注。现有策略通常为每项任务分配专属提示子集,虽有效但计算开销大,内存随任务数线性增长;而单一共享提示虽高效,却易受知识干扰影响表现。为此,本文提出SMoPE框架,融合任务专属与共享提示的优势。受前缀调优与专家混合(MoE)关系启发,将共享提示组织为稀疏MoE架构中的多个“提示专家”。对每个输入,仅激活少数相关专家,有效缓解干扰。为支持专家选择,引入提示注意力得分聚合机制,计算各专家统一代理得分,实现动态稀疏激活。此外,设计自适应噪声机制以促进专家均衡使用并保留旧任务知识。为增强专家专属性,提出基于原型的损失函数,利用前缀键作为隐式记忆表示。在多个持续学习基准上的实验表明,SMoPE持续优于任务专属提示方法,性能媲美当前最优方案,同时显著降低参数量和计算成本。
原文摘要 · Abstract (English)
Prompt-based methods have recently gained prominence in Continual Learning (CL) due to their strong performance and memory efficiency. A prevalent strategy in this paradigm assigns a dedicated subset of prompts to each task, which, while effective, incurs substantial computational overhead and causes memory requirements to scale linearly with the number of tasks. Conversely, approaches employing a single shared prompt across tasks offer greater efficiency but often suffer from degraded performance due to knowledge interference. To reconcile this trade-off, we propose SMoPE, a novel framework that integrates the benefits of both task-specific and shared prompt strategies. Inspired by recent findings on the relationship between Prefix Tuning and Mixture of Experts (MoE), SMoPE organizes a shared prompt into multiple "prompt experts" within a sparse MoE architecture. For each input, only a select subset of relevant experts is activated, effectively mitigating interference. To facilitate expert selection, we introduce a prompt-attention score aggregation mechanism that computes a unified proxy score for each expert, enabling dynamic and sparse activation. Additionally, we propose an adaptive noise mechanism to encourage balanced expert utilization while preserving knowledge from prior tasks. To further enhance expert specialization, we design a prototype-based loss function that leverages prefix keys as implicit memory representations. Extensive experiments across multiple CL benchmarks demonstrate that SMoPE consistently outperforms task-specific prompt methods and achieves performance competitive with state-of-the-art approaches, all while significantly reducing parameter counts and computational costs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。