通过低秩补偿实现专家模型的高效带宽利用与精度保持。
Bandwidth-Efficient Adaptive Mixture-of-Experts via Low-Rank Compensation
- 用预计算的低秩补偿因子动态恢复路由选择的专家精度。
- 仅传输前n个专家的紧凑低秩因子，其余保持低比特，减少通信开销。
- 适用于需要高吞吐和低延迟推理的异构部署场景。
Mixture-of-Experts（MoE）模型通过稀疏激活扩展容量，但对内存和带宽造成压力。离线卸载可缓解GPU内存负担，但逐标记路由导致不规则传输，使推理受I/O限制。静态统一量化虽减少流量，但在激进压缩下因忽略专家异质性而降低准确率。我们提出基于低秩补偿的带宽高效自适应专家混合模型，在推理时利用路由器引导进行精度恢复。每个标记仅传输前n（n<k）个专家的紧凑低秩因子，并对其应用补偿，其余专家保持低比特。集成于GPU与GPU-NDP系统，该方法在带宽与精度间取得更优平衡，显著提升吞吐量。
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) models scale capacity via sparse activation but stress memory and bandwidth. Offloading alleviates GPU memory by fetching experts on demand, yet token-level routing causes irregular transfers that make inference I/O-bound. Static uniform quantization reduces traffic but degrades accuracy under aggressive compression by ignoring expert heterogeneity. We present Bandwidth-Efficient Adaptive Mixture-of-Experts via Low-Rank Compensation, which performs router-guided precision restoration using precomputed low-rank compensators. At inference time, our method transfers compact low-rank factors with Top-n (n<k) experts per token and applies compensation to them, keeping others low-bit. Integrated with offloading on GPU and GPU-NDP systems, our method delivers a superior bandwidth-accuracy trade-off and improved throughput.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。