从预训练模型中高效提取专家网络，显著降低计算量。
Efficient Data Driven Mixture-of-Expert Extraction from Trained Networks
- 通过聚类激活输出，分两阶段提取专家子网络。
- 无需重训练，微调后达原模型98%性能，减少36%计算量。
- 适合资源受限场景的高效视觉模型部署。
视觉变换器在多种计算机视觉任务中表现卓越，但其高计算和资源需求带来挑战。虽然混合专家（MoE）可提升效率，但通常需昂贵的重训练甚至从头训练。近期研究利用预训练模型，发现编码器块中MLP层存在稀疏激活模式，可实现按样本条件激活相关子网络。本文提出新方法，从预训练模型的MLP层中后训练提取专家子网络，分为两阶段：首先聚类输出激活以识别不同模式；其次基于聚类结果提取对应子网络。在ImageNet-1k分类任务中，提取的专家网络可直接使用且表现优异，仅需少量微调即可恢复原模型98%性能，同时将乘加操作数（MACs）和模型大小分别降低最多36%和32%。
原文摘要 · Abstract (English)
Vision Transformers have emerged as the state-of-the-art models in various Computer Vision tasks, but their high computational and resource demands pose significant challenges. While Mixture-of-Experts (MoE) can make these models more efficient, they often require costly retraining or even training from scratch. Recent developments aim to reduce these computational costs by leveraging pretrained networks. These have been shown to produce sparse activation patterns in the Multi-Layer Perceptrons (MLPs) of the encoder blocks, allowing for conditional activation of only relevant subnetworks for each sample. Building on this idea, we propose a new method to construct MoE variants from pretrained models. Our approach extracts expert subnetworks from the model's MLP layers post-training in two phases. First, we cluster output activations to identify distinct activation patterns. In the second phase, we use these clusters to extract the corresponding subnetworks responsible for producing them. On ImageNet-1k recognition tasks, we demonstrate that these extracted experts can perform surprisingly well out of the box and require only minimal fine-tuning to regain 98% of the original performance, all while reducing MACs and model size, by up to 36% and 32% respectively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。