让手机也能流畅运行大模型，突破内存限制
FlexInfer: Breaking Memory Constraint via Flexible and Efficient Offloading for On-Device LLM Inference
- 通过异步预取、内存均衡锁定等技术动态调度计算资源
- 在有限内存下吞吐量最高提升12.5倍
- 适合移动端部署大模型的开发者与研究者
大型语言模型（LLMs）在设备端推理时面临高内存需求的挑战。传统降内存方法常以牺牲性能为代价，且缺乏适应性。本文提出FlexInfer，一种面向设备端推理的优化卸载框架，采用异步预取、平衡内存锁定和灵活张量保留等技术，提升内存效率并缓解I/O瓶颈，在用户指定资源约束下保持高性能。实验表明，该方法在资源受限条件下显著提升吞吐量，相比现有方法最高达12.5倍性能提升，推动大模型在资源受限设备上的部署。
原文摘要 · Abstract (English)
Large Language Models (LLMs) face challenges for on-device inference due to high memory demands. Traditional methods to reduce memory usage often compromise performance and lack adaptability. We propose FlexInfer, an optimized offloading framework for on-device inference, addressing these issues with techniques like asynchronous prefetching, balanced memory locking, and flexible tensor preservation. These strategies enhance memory efficiency and mitigate I/O bottlenecks, ensuring high performance within user-specified resource constraints. Experiments demonstrate that FlexInfer significantly improves throughput under limited resources, achieving up to 12.5 times better performance than existing methods and facilitating the deployment of large models on resource-constrained devices.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。