MNN-LLM让大模型在手机上跑得更快更省内存
MNN-LLM: A Generic Inference Engine for Fast Large Language Model Deployment on Mobile Devices
- 通过量化和内存混合存储降低内存占用
- 利用移动端指令集优化计算，速度提升8.6倍
- 适合想在手机上部署大模型的开发者
大型语言模型（LLMs）在多种任务中表现出色，但其巨大规模导致推理时计算资源消耗严重，成本高昂。因此，将推理部署到边缘设备成为可行方案。主要挑战在于内存使用和推理速度。本文提出MNN-LLM，一个专为移动设备加速大模型部署的通用推理引擎。该框架通过模型量化和DRAM-Flash混合存储策略，有效降低内存占用；基于移动端CPU指令集与GPU特性重排权重与输入，并采用多核负载均衡、混合精度浮点运算及几何计算等策略提升性能。实验表明，MNN-LLM相比当前主流专用框架，推理速度最高提升8.6倍。
原文摘要 · Abstract (English)
Large language models (LLMs) have demonstrated exceptional performance across a variety of tasks. However, their substantial scale leads to significant computational resource consumption during inference, resulting in high costs. Consequently, edge device inference presents a promising solution. The primary challenges of edge inference include memory usage and inference speed. This paper introduces MNN-LLM, a framework specifically designed to accelerate the deployment of large language models on mobile devices. MNN-LLM addresses the runtime characteristics of LLMs through model quantization and DRAM-Flash hybrid storage, effectively reducing memory usage. It rearranges weights and inputs based on mobile CPU instruction sets and GPU characteristics while employing strategies such as multicore load balancing, mixed-precision floating-point operations, and geometric computations to enhance performance. Notably, MNN-LLM achieves up to a 8.6x speed increase compared to current mainstream LLM-specific frameworks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。