让手机跑更大LLM，通过动态交换内存与闪存中的权重。
Scaling Up On-Device LLMs via Active-Weight Swapping Between DRAM and Flash
- 用激活值预判后续层权重，重叠计算与加载过程。
- 在有限内存下实现比现有方法更优的性能与成本平衡。
- 适合资源受限设备上部署大模型的开发者参考。
大型语言模型（LLMs）正越来越多地部署于移动设备，但受限于有限的DRAM容量，可部署的模型规模受到制约。本文提出ActiveFlow，首个适用于现代非ReLU型LLM的自适应DRAM使用推理框架，实现了内存与闪存间活跃权重的动态交换，从而扩大可部署模型尺寸。该框架包含三项创新技术：(1) 跨层活跃权重预加载，利用当前层激活预测后续几层活跃权重，实现计算与数据加载的重叠，并支持大块I/O传输；(2) 稀疏感知自蒸馏，调整活跃权重以匹配稠密模型输出分布，补偿上下文稀疏性带来的近似误差；(3) 活跃权重内存-闪存交换流水线，根据可用内存动态调度热权重缓存、预加载的活跃权重及计算中权重的空间分配。实验表明，ActiveFlow在性能与成本之间达到了优于现有优化方法的帕累托前沿。
原文摘要 · Abstract (English)
Large language models (LLMs) are increasingly being deployed on mobile devices, but the limited DRAM capacity constrains the deployable model size. This paper introduces ActiveFlow, the first LLM inference framework that can achieve adaptive DRAM usage for modern LLMs (not ReLU-based), enabling the scaling up of deployable model sizes. The framework is based on the novel concept of active weight DRAM-flash swapping and incorporates three novel techniques: (1) Cross-layer active weights preloading. It uses the activations from the current layer to predict the active weights of several subsequent layers, enabling computation and data loading to overlap, as well as facilitating large I/O transfers. (2) Sparsity-aware self-distillation. It adjusts the active weights to align with the dense-model output distribution, compensating for approximations introduced by contextual sparsity. (3) Active weight DRAM-flash swapping pipeline. It orchestrates the DRAM space allocation among the hot weight cache, preloaded active weights, and computation-involved weights based on available memory. Results show ActiveFlow achieves the performance-cost Pareto frontier compared to existing efficiency optimization methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。