优化边缘大模型冷启动延迟，让加载与计算并行进行
CSGO: Generalized Optimization for Cold Start in Wireless Collaborative Edge LLM Systems
- 动态调整模型分层与设备分配，隐藏加载时间
- 冷启动延迟显著降低，优于基线策略
- 适合资源受限的边缘AI部署场景
在边缘设备上部署大语言模型可实现低延迟、隐私保护的AI服务，但受限于设备资源。虽然流水线并行能支持分布式推理，现有方法常忽略因按需加载模型带来的冷启动延迟。本文提出一种感知延迟的调度框架，通过将模型加载与计算、通信重叠，最小化总推理延迟。基于设备和模型参数，框架动态调整层划分与分配，有效隐藏加载时间，尽可能消除空闲时段。我们将问题建模为混合整数非线性规划，并设计高效动态规划算法以优化模型分片与设备分配。实验表明，该方法相比基线策略显著降低了冷启动延迟。
原文摘要 · Abstract (English)
While deploying large language models on edge devices promises low-latency and privacy-preserving AI services, it is hindered by limited device resources. Although pipeline parallelism facilitates distributed inference, existing approaches often ignore the cold-start latency caused by on-demand model loading. In this paper, we propose a latency-aware scheduling framework that overlaps model loading with computation and communication to minimize total inference latency. Based on device and model parameters, the framework dynamically adjusts layer partitioning and allocation to effectively hide loading time, thereby eliminating as many idle periods as possible. We formulate the problem as a Mixed-Integer Non-Linear Program and design an efficient dynamic programming algorithm to optimize model partitioning and device assignment. Experimental results show that the proposed method significantly reduces cold-start latency compared to baseline strategies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。