提出Hermes框架，让大模型在边缘设备上更省内存、更快推理。
Hermes: Memory-Efficient Pipeline Inference for Large Models on Edge Devices
- 动态内存管理+并行加载，优化模型执行流程
- 比现有方案内存降低86.7%，速度提升4.24倍
- 适合资源受限的边缘设备部署大模型
近年来，基于Transformer的大规模模型应用取得了显著成功。然而，模型参数的指数级增长给边缘部署带来了巨大的内存挑战。以往工作主要聚焦于优化模型结构或采用内存交换方法，前者降低推理精度，后者增加推理延迟。本文提出PIPELOAD，一种新型的内存高效流水线执行机制，通过动态内存管理降低内存占用，并利用并行模型加载减少推理延迟。基于该机制，我们构建了专为边缘设备优化的大模型推理框架Hermes。我们在不同规模的Transformer模型上进行了评估，实验表明，Hermes在BERT和ViT模型上相比当前最优流水线机制，推理速度提升达4.24倍，内存消耗降低86.7%；在GPT类模型上，推理速度提升2.58倍，内存消耗降低90.3%。
原文摘要 · Abstract (English)
The application of Transformer-based large models has achieved numerous success in recent years. However, the exponential growth in the parameters of large models introduces formidable memory challenge for edge deployment. Prior works to address this challenge mainly focus on optimizing the model structure and adopting memory swapping methods. However, the former reduces the inference accuracy, and the latter raises the inference latency. This paper introduces PIPELOAD, a novel memory-efficient pipeline execution mechanism. It reduces memory usage by incorporating dynamic memory management and minimizes inference latency by employing parallel model loading. Based on PIPELOAD mechanism, we present Hermes, a framework optimized for large model inference on edge devices. We evaluate Hermes on Transformer-based models of different sizes. Our experiments illustrate that Hermes achieves up to 4.24 X increase in inference speed and 86.7% lower memory consumption than the state-of-the-art pipeline mechanism for BERT and ViT models, 2.58 X increase in inference speed and 90.3% lower memory consumption for GPT-style models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。