arXiv:2607.20427cs.CLcs.AI2026-07

MoE路由其实像哈夫曼编码,高频任务用短路径,低频复杂任务用长而多样的路径。

Is MoE Routing a Huffman Code? Discovering the Frequency-Diversity Law in Chain-of-Thought

论文配图:Is MoE Routing a Huffman Code? Discovering the Frequency-Diversity Law in Chain-of-Thought
图 1 · 摘自论文原文
  • 发现MoE路由自发遵循信息论中的哈夫曼编码规律,高频内容走短路径。
  • 在Phi-3.5-MoE等模型中,稀疏专家资源被用于常见词,罕见复杂推理调用多样专家组合。
  • 提出剪枝策略消除冗余,提升模型压缩效率,适合关注模型优化与可解释性的研究者。

Mixture-of-Experts架构推动了模型规模化,但其路由机制仍不透明。本文揭示一个根本性规律:MoE路由不仅是选择,更是哈夫曼编码的体现。我们提出频率-多样性定律,发现如Phi-3.5-MoE和Gemma-4-27B-A4B等先进模型自发扮演信息论引擎角色——对常见标记使用稀疏专家资源,对链式思维轨迹中的罕见、复杂任务则调用高多样性专家委员会。然而我们发现Qwen3.5-35B-A3B存在关键冗余陷阱:当有效稀疏度(k/E_eff)过低时,负载均衡反而引入功能冗余,掩盖了底层哈夫曼效率信号。为此,我们提出子集差值剪枝(Subset Difference Pruning),一种精准剔除功能重复项的策略。实验证明,剪枝不仅不损害推理能力,反而释放模型潜在的哈夫曼效率,迫使逻辑收敛为精简、高密度路径。研究建议下一代MoE应摆脱强制负载均衡,转向最小描述长度(MDL)最优——为高频信息分配短路由码,为低频信息分配长而多样的码,使路由从启发式设计转变为有原则的压缩引擎。

原文摘要 · Abstract (English)

Mixture-of-Experts architectures have revolutionized scaling, yet the underlying logic of their routing remains a black box. In this paper, we uncover a fundamental governing principle: MoE routing is not merely selection, but a manifestation of Huffman Coding. We introduce the Frequency-Diversity Law, revealing that state-of-the-art models, such as Phi-3.5-MoE and Gemma-4-27B-A4B, spontaneously act as information-theoretic engines. These models allocate sparse expert resources for common tokens while invoking high-diversity expert committees for rare, complex tasks found in chain-of-thought trajectories. However, we identify a critical redundancy trap in Qwen3.5-35B-A3B: when effective sparsity (k/E_eff) is sufficiently low, load-balancing inadvertently imposes functional redundancy, masking the underlying Huffman efficiency signal. To bridge this gap, we propose Subset Difference Pruning, a surgical strategy to eliminate functional duplicates. We demonstrate that pruning does not degrade reasoning; instead, it unleashes the model's latent Huffman efficiency, forcing the logic to collapse into streamlined, high-density paths. Our findings suggest that the next generation of MoEs should move beyond forced load-balancing toward Minimum Description Length (MDL) optimality, assigning shorter expert-routing codes to high-frequency information and longer, more diverse codes to low-frequency information, thereby transforming routing from a heuristic into a principled compression engine.

MoE哈夫曼编码推理优化信息论

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。