TileQ通过二维分块低秩量化，大幅降低MoE模型部署开销。
TileQ: Efficient Low-Rank Quantization of Mixture-of-Experts with 2D Tiling
- 用二维分块共享输入输出维度的低秩因子，实现高效量化
- 内存占用降低10倍，推理延迟降至5%仍保持顶尖性能
- 无需微调，适合追求轻量化部署的工程师
Mixture-of-Experts（MoE）模型通过稀疏激活专用专家实现卓越性能，但专家参数量巨大，部署面临挑战。低秩量化虽具潜力，但现有方法仍存在显著内存开销和推理延迟。为此，我们提出不依赖微调的后训练量化方法TileQ，采用2D分块结构化低秩量化，实现专家在输入与输出维度间低秩因子的共享。同时，设计高效的推理机制，将多个低秩专家计算融合为单次遍历，显著提升硬件利用率。实验表明，TileQ可将额外内存使用减少至1/10，推理延迟降至约5%，且保持最先进的准确率。
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) models achieve remarkable performance by sparsely activating specialized experts, yet their massive parameters in experts pose significant challenges for deployment. While low-rank quantization offers a promising route to compress MoE models, existing methods still incur nonnegligible memory overhead and inference latency. To address these limitations, we propose \textsc{TileQ}, a fine-tuning-free post-training quantization (PTQ) method that employs 2D-tiling structured low-rank quantization to share low-rank factors across both input and output dimensions of MoE experts. Furthermore, we introduce an efficient inference technique for \textsc{TileQ} that fuses multiple low-rank expert computations into a single-pass operation, significantly improving hardware utilization. Experiments show that \textsc{TileQ} cuts down additional memory usage up to 10$\times$ and reduces inference latency to $\sim$5\% while preserving state-of-the-art accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。