arXiv:2604.06798cs.LGcs.AI2026-04被引 2

让专家模型在二值化后仍高效准确,解决冗余与路由失真问题。

MoBiE: Efficient Inference of Mixture of Binary Experts under Post-Training Quantization

  • 通过联合SVD降冗余,结合梯度与海森矩阵优化权重重要性评估。
  • 在多个MoE模型上降低52.2%困惑度,零样本性能提升43.4%,推理速度翻倍。
  • 无额外存储开销,适合部署资源受限的二值化专家模型系统。

基于专家混合(MoE)的大语言模型虽性能优异,但存在高内存与计算开销。权重二值化可极大提升效率,但现有针对密集型大模型的二值化方法难以应对MoE特有的问题,如跨专家冗余、任务无关的重要性估计以及量化引起的路由偏移。为此,我们提出首个专为MoE大模型设计的二值化框架MoBiE。MoBiE包含三项核心创新:1)采用联合SVD分解降低跨专家冗余;2)将全局损失梯度融入局部海森矩阵,提升权重重要性评估精度;3)引入基于输入零空间的误差约束,缓解路由失真。值得注意的是,MoBiE在不增加额外存储开销的前提下实现优化,兼顾效率与性能。大量实验表明,MoBiE在多个MoE大模型及基准测试中持续优于当前最优二值化方法。例如,在Qwen3-30B-A3B上,困惑度降低52.2%,平均零样本性能提升43.4%,推理速度提升超2倍,并进一步缩短量化时间。代码已公开于https://github.com/Kishon-zzx/MoBiE。

原文摘要 · Abstract (English)

Mixture-of-Experts (MoE) based large language models (LLMs) offer strong performance but suffer from high memory and computation costs. Weight binarization provides extreme efficiency, yet existing binary methods designed for dense LLMs struggle with MoE-specific issues, including cross-expert redundancy, task-agnostic importance estimation, and quantization-induced routing shifts. To this end, we propose MoBiE, the first binarization framework tailored for MoE-based LLMs. MoBiE is built on three core innovations: 1. using joint SVD decomposition to reduce cross-expert redundancy; 2. integrating global loss gradients into local Hessian metrics to enhance weight importance estimation; 3. introducing an error constraint guided by the input null space to mitigate routing distortion. Notably, MoBiE achieves these optimizations while incurring no additional storage overhead, striking a balance between efficiency and model performance. Extensive experiments demonstrate that MoBiE consistently outperforms state-of-the-art binary methods across multiple MoE-based LLMs and benchmarks. For example, on Qwen3-30B-A3B, MoBiE reduces perplexity by 52.2$\%$, improves average zero-shot performance by 43.4$\%$, achieves over 2 $\times$ inference speedup, and further shortens quantization time. The code is available at https://github.com/Kishon-zzx/MoBiE.

专家模型二值化高效推理量化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。