动态路由提升大模型推理质量与速度平衡
Dynamic Quality-Latency Aware Routing for LLM Inference in Wireless Edge-Device Networks
- 根据任务复杂度动态分配计算任务到手机或边缘服务器
- 在保持推理质量前提下，延迟降低5-15%，调用次数减少10-20%
- 适用于需要实时响应的移动端大模型应用
无线通信与大语言模型（LLMs）的融合将推动无处不在的智能服务，但在无线边缘设备协同环境中部署时，推理质量与端到端延迟之间存在关键权衡。任务复杂度与资源分配不匹配：简单查询外发导致延迟过高，而本地模型又难以处理复杂计算。为此，我们提出一种动态、质量-延迟感知的路由框架，协调移动端轻量模型与边缘服务器强大模型之间的推理。针对单轮查询，融合BERT预测的语义得分与通信、计算开销；针对多轮对话，进一步量化因模型切换和KV缓存管理带来的上下文相关成本。在保持全推理质量的前提下，大量实验表明，该框架在MMLU、GSM8K和MT-Bench-101基准上，平均响应延迟降低5-15%，大型模型调用次数减少10-20%。
原文摘要 · Abstract (English)
The integration of wireless communications and Large Language Models (LLMs) is poised to unlock ubiquitous intelligent services, yet deploying them in wireless edge-device collaborative environments presents a critical trade-off between inference quality and end-to-end latency. A fundamental mismatch exists between task complexity and resource allocation: offloading simple queries invites prohibitive latency, while on-device models lack the capacity for demanding computations. To address this challenge, we propose a dynamic, quality-latency aware routing framework that orchestrates inference between a lightweight model on the mobile device and a powerful model on the edge server. Our framework employs two distinct cost models: for single-turn queries, it fuses a BERT-predicted semantic score with communication and computation overheads; for multi-turn dialogues, it further quantifies context-aware costs arising from model switching and KV-cache management. While maintaining full inference quality, extensive experiments demonstrate that our framework cuts average response latency by 5-15% and reduces large model invocations by 10-20% against competitive baselines on MMLU, GSM8K, and MT-Bench-101 benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。