通过拓扑分析发现专家间隐藏的三元冲突,实现更优的稀疏专家模型压缩。
HodgeCover: Higher-Order Topological Coverage Drives Compression of Sparse Mixture-of-Experts

- 利用拓扑学中的霍奇分解识别专家间的三元不可合并结构
- 在三个开源MoE模型上实现比现有方法更激进的压缩率,保持性能
- 适合关注模型压缩与结构优化的研究者和工程师
稀疏混合专家(MoE)层将输入分发至少数专家,无需重新训练即可实现学习无关的压缩。然而,现有压缩方法均受制于一个微妙障碍:三个专家两两兼容,但合并时可能形成不可约的循环,任何仅基于成对信号评分的方法都无法识别哪些三元组可联合合并。本文揭示该障碍是2-复形中单纯形拉普拉斯算子的调和核,其顶点为专家,边携带KL合并屏障,面携带三元屏障;通过对边屏障信号进行霍奇分解,可精确分离出该核。我们据此设计选择目标:HodgeCover贪心覆盖调和关键边与三元关键三角形,并与幸存者的现成权重剪枝结合。在三个公开权重的稀疏MoE主干网络下,面对激进的专家缩减,HodgeCover在专家缩减轴上达到当前最优水平,在混合压缩轴上领先,且唯一实现了所有四个霍奇分量的保留质量均衡。结果表明,暴露学习到的MoE结构的调和核,能改变最相关压缩场景中胜出的压缩器。
原文摘要 · Abstract (English)
Sparse Mixture-of-Experts (MoE) layers route tokens through a handful of experts, and learning-free compression of these layers reduces inference cost without retraining. A subtle obstruction blocks every existing compressor in this family: three experts can each be pairwise compatible yet form an irreducible cycle when merged together, so any score that ranks experts on pairwise signals is structurally blind to which triples are jointly mergeable. We show the obstruction is a precise mathematical object, the harmonic kernel of the simplicial Laplacian on a 2-complex whose vertices are experts, whose edges carry KL merge barriers, and whose faces carry triplet barriers; Hodge-decomposing the edge-barrier signal isolates the kernel exactly. We turn the diagnostic into a selection objective: HodgeCover greedily covers the harmonic-critical edges and triplet-critical triangles, and a hybrid variant of HodgeCover pairs it with off-the-shelf weight pruning on survivors. On three open-weight Sparse MoE backbones under aggressive expert reduction, HodgeCover matches state-of-the-art learning-free baselines on the expert-reduction axis, leads on the aggressive-compression frontier of the hybrid axis, and uniquely balances retained mass across all four Hodge components. These results show that exposing the harmonic kernel of a learned MoE structure changes which compressor wins at the regime that matters most.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。