将大模型拆分到多设备协同推理，让低功耗设备也能运行超大语言模型。
Model-Distributed Inference for Large Language Models at the Edge
- 把大模型分块部署在多个边缘设备上，通过设备间传递中间结果协同计算。
- 支持并行生成多条文本，设备越多吞吐量越高，单机内存占用越低。
- 适合资源受限的边缘场景，如智能终端、物联网设备部署大模型。
我们提出面向边缘端大型语言模型（LLM）的模型分布式推理框架MDI-LLM，通过将模型划分为多个分区并分配至网络中的不同设备，实现跨低功耗设备的部署。各设备通过设备间链路交换中间激活向量，协同完成计算。为提升效率，引入“循环流水线并行”技术，减少设备空闲时间，支持多序列并行生成。借助多设备联合算力，MDI-LLM可运行超出单个设备内存容量的模型，在低成本硬件上实现推理。随着参与设备增加，令牌生成吞吐量提升，单设备内存消耗降低。
原文摘要 · Abstract (English)
We introduce Model-Distributed Inference for Large-Language Models (MDI-LLM), a novel framework designed to facilitate the deployment of state-of-the-art large-language models (LLMs) across low-power devices at the edge. This is accomplished by dividing the model into multiple partitions, which are then assigned to different devices/nodes within the network. These nodes exchange intermediate activation vectors via device-to-device links, enabling collaborative computation. To enhance the efficiency of this process, we propose the "recurrent pipeline parallelism" technique, which reduces idle time on each device and facilitates parallel inference during the generation of multiple text sequences. By leveraging the combined computational resources of multiple edge devices, MDI-LLM enables the deployment of LLMs that exceed the memory capacity of individual devices, making it possible to perform inference on low-cost hardware. Furthermore, as the number of participating devices increases, MDI-LLM boosts token generation throughput and reduces memory consumption per device.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。