将冻结的ViT转为稀疏专家模型,不更新主干权重仍保持高精度。
CLEAR-MoE: Shared-Basis Expert Extraction from Frozen Vision Transformers via Calibration-Driven Layer Selection

- 通过分层评分与聚类,提取共享低秩基和各簇专家
- 在多个模型上保留99.9%准确率,误差不超过0.1个百分点
- 适合想轻量化部署视觉Transformer的研究者
我们提出CLEAR-MoE,一种四阶段后训练流程,将冻结的预训练视觉变换器(ViT)转化为稀疏混合专家(MoE)模型,无需更新主干权重。该流程首先基于稀疏性、可聚类性和输出敏感性对前馈网络(FFN)层进行评分;其次利用k-means聚类将选中层分解为共享低秩SVD基和每簇残差专家;再以聚类标签监督训练轻量级路由器;最后通过可插拔CUDA后端调度令牌。在Imagenette数据集上使用DeiT-Small时,CLEAR-MoE准确率达86.70±0.02%,仅比密集模型(86.73%)低0.03个百分点。大量消融实验表明,共享SVD基是保持精度的关键因素:随机路由、学习路由及三种不同路由器架构性能相近,准确率波动不超过0.06个百分点(86.62%–86.68%)。准确率在不同SVD秩、专家数量(2–8)、校准集大小(50–500)及随机种子下均稳定。该方法泛化至五种ViT主干(DeiT-Tiny、DeiT-Small、DeiT-Base、ViT-Small、ViT-Base),参数量覆盖5.7M至86.6M,与密集模型的准确率差异≤0.10个百分点。在GTX 960 GPU上,路由与散收操作使CLEAR-MoE FFN比密集实现慢1.3–1.7倍。微基准测试显示,路由比专家矩阵乘法更受内存限制,提示融合调度核函数是未来优化方向。
原文摘要 · Abstract (English)
We present CLEAR-MoE, a four-phase post-training pipeline that converts a frozen pretrained Vision Transformer (ViT) into a sparse Mixture-of-Experts (MoE) model without updating backbone weights. The pipeline (i) scores feed-forward network (FFN) layers by sparsity, clusterability, and output sensitivity; (ii) decomposes selected layers into a shared low-rank SVD basis and per-cluster residual experts using k-means clustering; (iii) trains lightweight routers supervised by cluster labels; and (iv) dispatches tokens through pluggable CUDA backends. On Imagenette with DeiT-Small, CLEAR-MoE retains 99.9% of the dense model's accuracy (86.70 +/- 0.02% versus 86.73%). Extensive ablation studies reveal a consistent empirical finding: the shared SVD basis is the primary factor responsible for preserving accuracy. Random routing, learned routing, and three different router architectures produce nearly identical performance, with accuracy varying by at most 0.06 percentage points (86.62%-86.68%). Accuracy also remains stable across different SVD ranks, expert counts (2-8), calibration set sizes (50-500), and random seeds. This behavior generalizes across five ViT backbones (DeiT-Tiny, DeiT-Small, DeiT-Base, ViT-Small, and ViT-Base), covering models from 5.7M to 86.6M parameters, with accuracy differences <= 0.10 percentage points from their dense counterparts. On a GTX 960 GPU, routing and scatter-gather overhead make the CLEAR-MoE FFN 1.3-1.7x slower than the dense implementation. A dispatch microbenchmark further shows that routing is an order of magnitude more memory-bound than expert matrix multiplications, identifying fused dispatch kernels as a promising direction for future optimization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。