让大模型在物联网设备上高效运行,解决内存与延迟瓶颈。
Memory- and Latency-Constrained Inference of Large Language Models via Adaptive Split Computing
- 按生成过程动态分割模型,分段采用不同精度压缩
- 通信开销降低,推理速度提升1.49倍,准确率不降反升
- 专为边缘设备设计,适合资源受限场景部署
大语言模型在多种推理任务中已接近人类水平,但其在资源受限的物联网设备上部署仍不可行,主要受限于庞大的参数量和内存密集型的自回归解码。尽管分割计算可通过将模型执行分布在边缘设备与云服务器间提供解决方案,但现有方法未能解决自回归推理的独特挑战,尤其是迭代式令牌生成过程和不断增长的键值缓存需求。本文提出首个专为边缘设备部署设计的自回归感知分割计算框架。主要贡献包括:首先,提出单点分割压缩(OPSC),一种混合精度量化方案,通过将模型战略性地划分为前端和后端,以不同精度运行,防止内存溢出;其次,设计两阶段中间压缩管道,结合阈值分割(TS)与逐令牌自适应比特量化(TAB-Q),在保留关键激活的同时大幅减少通信开销;第三,构建统一优化框架,联合选择最优分割点、量化配置和序列长度,满足严格的内存与延迟约束。在多种大模型和硬件平台上的广泛评估表明,该框架优于当前主流量化方法(如SmoothQuant、OmniQuant、Atom),实现1.49倍的推理加速,并显著降低通信开销,同时保持或提升模型准确率。
原文摘要 · Abstract (English)
Large language models (LLMs) have achieved near-human performance across diverse reasoning tasks, yet their deployment on resource-constrained Internet-of-Things (IoT) devices remains impractical due to massive parameter footprints and memory-intensive autoregressive decoding. While split computing offers a promising solution by partitioning model execution between edge devices and cloud servers, existing approaches fail to address the unique challenges of autoregressive inference, particularly the iterative token generation process and expanding key-value (KV) cache requirements. This work introduces the first autoregressive-aware split computing framework designed explicitly for LLM deployment on edge devices. Our approach makes three key contributions. First, we develop one-point split compression (OPSC), a mixed-precision quantization scheme that prevents out-of-memory failures by strategically partitioning models into front-end and back-end segments with different precision levels. Second, we propose a two-stage intermediate compression pipeline that combines threshold splitting (TS) and token-wise adaptive bit quantization (TAB-Q) to preserve accuracy-critical activations while dramatically reducing communication overhead. Third, we formulate a unified optimization framework that jointly selects optimal split points, quantization settings, and sequence lengths to satisfy strict memory and latency constraints. Extensive evaluations across diverse LLMs and hardware platforms demonstrate superior performance compared to state-of-the-art quantization methods, including SmoothQuant, OmniQuant, and Atom. The framework achieves a 1.49 inference speedup and significant communication overhead reduction while maintaining or improving model accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。