让大模型路由更懂延迟，兼顾速度、质量与成本。
Beyond Accuracy and Cost: Latency-Aware LLM Query Routing for Dynamic Workloads
- 用轻量级延迟估计算法预测生成首字时间。
- 联合优化延迟、准确率与成本，提升40%效率。
- 适合高并发动态负载下的大模型服务部署。
现代语言查询路由器通过将查询分配给在响应质量与成本间平衡的模型来提升推理效率。然而，现有路由器大多忽略延迟，不考虑查询在模型实例上的生成延迟。实际中，延迟常由轮询或短队列优先等负载均衡策略控制，这些策略未考虑模型准确率或推理成本。将延迟纳入路由决策极具挑战性，因为延迟不仅取决于提示长度，还受模型实例的预填充与解码负载、服务框架的调度与批处理策略影响。本文设计了一种轻量级延迟估计算法，模拟服务框架中的自回归分批处理过程，估算查询的首字时间（TTFT）。将该估计算法集成至延迟感知路由器中，联合优化延迟、准确率与成本。实验表明，这种联合优化可使准确率-成本效用提升最高达40%，同时保持与标准负载均衡方法相当的延迟水平。
原文摘要 · Abstract (English)
Modern language query routers improve inference efficiency by assigning each query to a model that balances response quality and monetary cost. However, current query routers are largely latency-agnostic and do not consider the generation latency experienced by queries at model instances. In practice, latency is often controlled by load-balancing policies such as round-robin or join-the-shortest-queue, which do not account for model accuracy or inference cost. Incorporating query latency into routing is challenging as it depends not only on the query's prompt length, but also on the current prefill and decode workload at the model instance and the scheduling and batching policy of the serving framework. We design a lightweight latency estimator that simulates autoregressive token batch processing in the serving framework and estimates the time-to-first-token (TTFT) of queries. We incorporate this latency estimator into a latency-aware router that jointly optimizes latency, accuracy, and cost when assigning queries to model instances. Our experimental results indicate that this joint optimization yields up to 40% improvement in accuracy--cost utility while maintaining the same latencies as standard load-balancing approaches.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。