将大模型专家混合架构压缩至手机端运行，解决内存与延迟瓶颈。
Tiny-QMoE
- 针对移动端8GB以下内存设计轻量化MoE压缩方法
- 在保持高稀疏性的同时实现模型体积缩小至原1/10
- 适合边缘设备离线部署，无需依赖云端服务器
QMoE模型为大规模专家混合（MoE）模型的压缩提供了实用方案，尤其适用于内存受限场景，如移动设备仅有4-8GB统一内存且需共享系统资源。尽管当前主流硬件如H100/V100拥有80GB HBM，但移动端计算能力仍远低于服务器。该研究关注在低功耗、低延迟环境下运行大模型的需求，提出仅考虑MoE结构的压缩策略，避免对非专家混合系统进行评估。重点在于克服远程请求延迟问题，支持无网络连接环境下的本地推理，使LLM可在手机等边缘设备上高效运行。
原文摘要 · Abstract (English)
The QMoE model provides a practical approach for compression of massive Mixture-of-Experts (MoE) models. QMoE offers a solution geared towards memory limitations that often reach terabyte scales, and it has the advantage of working with high sparsity models which implicitly lend themselves to compression techniques. QMoE also has the advantage of only taking MoE models into account and does not evaluate its use with non mixture of expert systems. Although this prior attempt focuses on the limitations of large servers with the latest NVIDIA hardware which in the case of the H100 and V100 which have 80 GB of HBM (High Bandwidth Memory), what is not being considered is a significantly more constrained environment, such as in the case of mobile devices which may have in the case of the iPhone anywhere from 4 to 8 GB of unified memory which also needs to be shared with the operating system and additional processes. Although edge devices such as phones and laptops are becoming increasingly more computationally powerful, they are still not close to the level of advanced server machines such as NVIDIA. An additional constraint that we must consider is that of latency. The communication time of sending a request to an LLM server and then getting it back is an additional waiting time that can be removed. We may also want to use LLM technology in environments where there is no reliable network connection.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。