边端部署大模型，联合量化与分层推理降延迟。
DILEMMA: Joint LLM Quantization and Distributed LLM Inference Over Edge Computing Systems
- 联合优化模型分层部署与逐层量化策略。
- 在OPT-350上实现12.75%量化率且模型损失不变。
- 适合资源受限的边缘计算场景，如智慧城市应用。
随着大型语言模型（LLMs）在智慧城市建设中广泛应用，需将其推向网络边缘以降低响应延迟。边缘计算（EC）作为靠近终端用户的计算资源，可减少服务请求的通信开销，但其在通信、计算和存储方面能力有限。本文提出DILEMMA框架，通过联合优化层位置分配与层量化，在保证模型性能的前提下最小化总推理延迟。该框架采用逐层量化与知识蒸馏技术控制性能损失，并构建整数线性规划模型求解。在SQuAD数据集上对OPT-350模型的实验表明，DILEMMA实现了高达12.75%的量化比率且模型损失保持不变，验证了其在资源受限环境下的有效性。
原文摘要 · Abstract (English)
With a recent trend of using Large Language Models (LLMs) for different applications within smart cities, there is a need for pushing these models toward the edge of network while still preserving their performance. Edge Computing (EC) as a physically closer computing resource to the end users can help to reduce the communication delay for serving end users' tasks for LLM-dependent services. However, EC servers have limited capacity in terms of communication, computation, and storage capacity. This paper introduces DILEMMA, a novel framework addressing the challenges of deploying LLMs in EC systems by jointly optimizing layer placement and layer quantization in EC systems. DILEMMA formulates an Integer Linear Programming problem to minimize total inference delay while ensuring acceptable LLM performance levels, leveraging layer-wise quantization and knowledge distillation for LLM performance control. Experimental evaluations on OPT-350 model using the SQuAD dataset demonstrate that DILEMMA achieves a quantization ratio of up to 12.75% while preserving model loss, highlighting its effectiveness in resource-constrained environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。