让专家分工更明确，提升模型效率与能力。
CoMoE: Contrastive Representation for Mixture-of-Experts in Parameter-Efficient Fine-tuning
- 通过对比学习让激活与未激活专家差异更大
- 在多任务测试中显著提升专家模块化程度
- 适合需要高效微调的多任务场景
在参数高效微调中，混合专家（MoE）通过将功能分配给不同专家并稀疏激活，成为平衡模型容量与计算开销的有力方法。然而现有MoE变体在异构数据集上表现不佳，因专家可能学习相似知识，导致容量利用不足。本文提出对比表示混合专家（CoMoE），通过在top-k路由中对激活与未激活专家采样并引入对比目标，促进专家间差异化。实验证明该方法能有效弥补输入与两类专家间的互信息差距。在多个基准和多任务设置中，CoMoE可持续增强MoE能力并推动专家模块化。
原文摘要 · Abstract (English)
In parameter-efficient fine-tuning, mixture-of-experts (MoE), which involves specializing functionalities into different experts and sparsely activating them appropriately, has been widely adopted as a promising approach to trade-off between model capacity and computation overhead. However, current MoE variants fall short on heterogeneous datasets, ignoring the fact that experts may learn similar knowledge, resulting in the underutilization of MoE's capacity. In this paper, we propose Contrastive Representation for MoE (CoMoE), a novel method to promote modularization and specialization in MoE, where the experts are trained along with a contrastive objective by sampling from activated and inactivated experts in top-k routing. We demonstrate that such a contrastive objective recovers the mutual-information gap between inputs and the two types of experts. Experiments on several benchmarks and in multi-task settings demonstrate that CoMoE can consistently enhance MoE's capacity and promote modularization among the experts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。