arXiv:2603.23436cs.LG2026-03

针对小样本与任务重叠的持续学习难题,提出相似性感知专家混合模型。

Similarity-Aware Mixture-of-Experts for Data-Efficient Continual Learning

  • 通过渐进式全局池化与实例级提示掩码实现任务相似性感知
  • 在低数据量下仍保持良好性能,跨任务迁移效率提升显著
  • 适合数据稀缺且任务存在任意重叠的实际场景

机器学习模型在部署后常需适应新数据,以应对现实世界中的结构化或非结构化动态变化。持续学习(CL)框架支持模型持续更新,但现有方法多假设每个任务拥有充足数据或任务间无重叠。本文解决更普遍的情况:每个任务数据有限,且任务间可能存在任意重叠且无先验知识。该设定更具挑战性:一方面,数据稀缺要求有效利用通用知识并高效跨任务迁移;另一方面,非结构化任务重叠易引发负向知识迁移。为此,我们提出基于预训练模型的自适应专家混合(MoE)框架,逐步建立任务间的相似性感知。设计包含两个创新组件:增量全局池化和实例级提示掩码。前者通过时间渐进引入提示缓解提示关联噪声;后者将输入样本分解为与当前提示一致(分布内)和需新提示(分布外)两类。二者协同作用,在利用任务重叠潜力的同时,主动防止因每任务数据稀缺导致的负向干扰。在不同数据量与任务相似度下的实验表明,该方法显著提升样本效率,具有广泛适用性。

原文摘要 · Abstract (English)

Machine learning models often need to adapt to new data after deployment due to structured or unstructured real-world dynamics. The Continual Learning (CL) framework enables continuous model adaptation, but most existing approaches either assume each task contains sufficiently many data samples or that the learning tasks are non-overlapping. In this paper, we address the more general setting where each task may have a limited dataset, and tasks may overlap in an arbitrary manner without a priori knowledge. This general setting is substantially more challenging for two reasons. On the one hand, data scarcity necessitates effective contextualization of general knowledge and efficient knowledge transfer across tasks. On the other hand, unstructured task overlapping can easily result in negative knowledge transfer. To address the above challenges, we propose an adaptive mixture-of-experts (MoE) framework over pre-trained models that progressively establishes similarity awareness among tasks. Our design contains two innovative algorithmic components: incremental global pooling and instance-wise prompt masking. The former mitigates prompt association noise through gradual prompt introduction over time. The latter decomposes incoming task samples into those aligning with current prompts (in-distribution) and those requiring new prompts (out-of-distribution). Together, our design strategically leverages potential task overlaps while actively preventing negative mutual interference in the presence of per-task data scarcity. Experiments across varying data volumes and inter-task similarity show that our method enhances sample efficiency and is broadly applicable.

持续学习专家混合小样本

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。