通过语义一致性评估,智能分配大模型推理任务,显著降延迟省成本。
ConsRoute:Consistency-Aware Adaptive Query Routing for Cloud-Edge-Device Large Language Models
- 用重排序器直接衡量不同层级模型输出的语义一致性,生成精细路由信号。
- 复用预填充阶段隐藏状态,避免额外计算,设备端开销极低。
- 基于贝叶斯优化动态调整路由阈值,适配不同查询分布,兼顾质量与效率。
大语言模型虽能力强大,但推理延迟高、成本大,难以在时延敏感和资源受限场景部署。云-边-端协同推理通过动态调度查询至不同算力层级的模型,成为可行方案。本文提出ConsRoute,一种轻量、语义感知、自适应的路由框架,在最小化响应质量损失的前提下大幅提升推理效率。不同于以往依赖粗粒度质量预测的方法,ConsRoute利用重排序器直接评估跨层级模型输出间的语义一致性,生成细粒度软监督信号。为降低设备侧开销,该框架复用LLM预填充阶段的隐藏状态作为紧凑查询表征,避免额外编码器或推理步骤。这些表征被聚类后,采用贝叶斯优化学习各簇专属的路由阈值,实现异构查询分布下的质量、延迟与成本动态平衡。大量实验表明,ConsRoute在端到端延迟和推理成本上均降低近40%,且保持不低于95%的接近云端性能,持续优于现有基线方法。
原文摘要 · Abstract (English)
Large language models (LLMs) deliver impressive capabilities but incur substantial inference latency and cost, which hinders their deployment in latency-sensitive and resource-constrained scenarios. Cloud-edge-device collaborative inference has emerged as a promising paradigm by dynamically routing queries to models of different capacities across tiers. In this paper, we propose ConsRoute, a lightweight, semantic-aware, and adaptive routing framework that significantly improves inference efficiency while minimizing impact on response quality. Unlike prior routing methods that rely on predicting coarse-grained output quality gaps, ConsRoute leverages a reranker to directly assess the semantic consistency between responses generated by models at different tiers, yielding fine-grained soft supervision signals for routing. To minimize device-side overhead, ConsRoute reuses hidden states from the LLM prefilling stage as compact query representations, avoiding additional encoders or inference passes. Furthermore, these representations are clustered, and Bayesian optimization is employed to learn cluster-specific routing thresholds that dynamically balance quality, latency, and cost under heterogeneous query distributions. Extensive experiments demonstrate that ConsRoute achieves near-cloud performance (>=95%) while reducing end-to-end latency and inference cost by nearly 40%, consistently outperforming existing routing baselines in both response quality and system efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。