提出R²MoE框架，实现高效持续学习新视觉概念。
R^2MoE: Redundancy-Removal Mixture of Experts for Lifelong Concept Learning
- 用路由蒸馏机制让专家专注特定概念，防止遗忘。
- 移除冗余专家层，参数量减少63.3%。
- 适合需要长期学习新概念的个性化生成场景。
让大规模生成模型持续学习新视觉概念对个性化预训练模型至关重要。现有方法受限于灾难性遗忘和参数膨胀两大挑战。本文提出冗余消除混合专家（R²MoE）框架，实现参数高效的终身视觉概念学习。该框架包含三项创新：第一，设计带有路由蒸馏机制的混合专家结构，使专家获取概念专属知识的同时保留门控网络的路由能力，有效缓解灾难性遗忘；第二，提出消除逐层冗余专家的策略，充分复用已有专家，显著减少专家参数；第三，采用分层局部注意力引导推理，降低生成概念间的干扰。大量实验表明，相比最先进方法，本方法在概念保真度上表现更优，在CustomConcept 101数据集上遗忘率降低87.8%，参数量减少63.3%。代码已公开于https://github.com/learninginvision/R2MoE。
原文摘要 · Abstract (English)
Enabling large-scale generative models to continuously learn new visual concepts is essential for personalizing pre-trained models to meet individual user preferences. Existing approaches for continual visual concept learning are constrained by two fundamental challenges: catastrophic forgetting and parameter expansion. In this paper, we propose Redundancy-Removal Mixture of Experts (R^2MoE), a parameter-efficient framework for lifelong visual concept learning that effectively learns new concepts while incurring minimal parameter overhead. Our framework includes three key innovative contributions: First, we propose a mixture-of-experts framework with a routing distillation mechanism that enables experts to acquire concept-specific knowledge while preserving the gating network's routing capability, thereby effectively mitigating catastrophic forgetting. Second, we propose a strategy for eliminating redundant layer-wise experts that reduces the number of expert parameters by fully utilizing previously learned experts. Third, we employ a hierarchical local attention-guided inference approach to mitigate interference between generated visual concepts. Extensive experiments have demonstrated that our method generates images with superior conceptual fidelity compared to the state-of-the-art (SOTA) method, achieving an impressive 87.8\% reduction in forgetting rates and 63.3\% fewer parameters on the CustomConcept 101 dataset. Our code is available at {https://github.com/learninginvision/R2MoE}
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。