将大模型专家库合并压缩,实现高效部署且性能几乎不变。
UniMoMo: Expert Merging-Based MoE Acceleration for Large Recommendation Models

- 基于功能相似性合并专家,不依赖参数距离
- 4专家压缩后准确率超源模型,加速1.28倍至1.63倍
- 适合资源受限场景的推荐系统部署
稀疏混合专家(MoE)层通过条件计算扩展推荐能力,但训练好的模型仍需存储并路由全部专家。本文提出UniMoMo,一种后训练压缩框架,将模型转换为在显式专家预算下的更小标准MoE,无需引入压缩专用在线模块。该方法将问题建模为约束图粗化问题,依据未标注校准集上专家对共享推荐状态的响应相似性进行聚类,而非参数距离。为防止性能下降,引入层自适应保护机制,限制高流量专家的合并。在Amazon Beauty、KuaiRec和TenRec数据集上,含2、4、6个MoE块的模型经压缩至4专家后,平均NDCG@10相对原模型达99.92%–102.30%,A100加速1.28×–1.63×;极端压缩至2专家、top-1运行时,性能比为98.36%–104.24%,加速1.47×–2.21×。结果验证了完整转换与适配流程的有效性,表明训练好的推荐模型可按不同服务预算导出。
原文摘要 · Abstract (English)
Sparse mixture-of-experts (MoE) layers expand recommendation capacity through conditional computation, yet a trained checkpoint still stores and routes over its full expert bank. We study a deployment problem: convert that checkpoint to a smaller standard MoE under an explicit expert budget, without adding a compression-specific online module. To address this, we introduce UniMoMo, a post-training compression framework formulated as a constrained graph coarsening problem. Rather than relying on parameter distance, UniMoMo groups experts based on their functional similarity, using an unlabeled calibration set to measure how similarly experts respond to shared recommendation states. To prevent performance degradation, we introduce a layer-adaptive protection mechanism that restricts the merging of high-traffic experts based on their routing exposure. Across Amazon Beauty, KuaiRec, and TenRec with 2, 4, and 6 MoE blocks, the final four-expert checkpoints obtain source-relative five-run mean NDCG@10 ratios of 99.92%--102.30% and measured A100 speedups of 1.28$\times$--1.63$\times$. An aggressive two-expert, top-1 operating point obtains ratios of 98.36%--104.24% and speedups of 1.47$\times$--2.21$\times$. These endpoint results evaluate the complete conversion-and-adaptation workflow and show that a trained recommendation MoE can be exported at multiple serving budgets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。