动态调整专家网络,让模型持续学习不冗余也不过载。
Towards Adaptive Continual Model Merging via Manifold-Aware Expert Evolution

- 基于流形几何自动演化专家,避免盲目扩容
- 无需训练即可激活专家,准确率更高且节省参数
- 适合长期持续学习的轻量级模型更新场景
持续模型融合(CMM)在不进行大规模重训练的情况下,将任务特定模型逐步整合到统一架构中。然而,现有方法面临根本性困境:以主干为中心的方法受限于固定容量,易出现参数饱和与表示干扰;而混合专家(MoE)变体则盲目扩张,导致专家冗余和依赖额外数据优化的路由瓶颈。为此,我们提出 MADE-IT(流形感知的动态专家演化与隐式路由),通过将专家表示嵌入流形几何来实现专家管理与激活的自适应调控。引入基于投影的子空间亲和度度量与分布感知的自适应阈值机制,实现多样性与架构精简的平衡。此外,设计无需参数门控、无须训练的隐式路由机制,通过特征-子空间对齐激活专家。大量实验表明,MADE-IT 在长序列与乱序任务设置下均显著优于强基线,在准确率与鲁棒性上表现优异,同时大幅减少冗余专家,尤其在通用模块与早期层中效果显著。
原文摘要 · Abstract (English)
Continual Model Merging (CMM) sequentially integrates task-specific models into a unified architecture without intensive retraining. However, existing CMM methods are hindered by a fundamental saturation-redundancy dilemma: backbone-centric approaches face parameter saturation and representation interference within fixed capacities, whereas Mixture-of-Experts (MoE) variants resort to indiscriminate expansion, incurring expert redundancy and a routing bottleneck reliant on additional data-driven optimization. To resolve these challenges, we propose MADE-IT (Manifold-Aware Dynamic Expert Evolution and Implicit rouTing), an adaptive CMM method that orchestrates expert management and activation by grounding intrinsic expert representations in manifold geometry. We introduce a projection-based subspace affinity metric coupled with a distribution-aware adaptive threshold mechanism to guide autonomous expert evolution, harmonizing diversity with architectural parsimony. Furthermore, to bypass parameterized gating networks, we design a data-free and training-free implicit routing mechanism that activates experts via feature-subspace alignment. Extensive experiments demonstrate that MADE-IT consistently outperforms strong baselines in accuracy and robustness across long-horizon and shuffled task sequences, while significantly pruning redundant experts, particularly within generic modules and early layers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。