无需训练的在线路由算法，大幅提升多大模型服务效率
Efficient Training-Free Online Routing for High-Volume Multi-LLM Serving
- 基于近似最近邻搜索估算查询特征，一次优化生成全局路由策略
- 理论证明可达到近似最优性能，实测吞吐提升4.25倍
- 适合高并发、低预算场景，代码开源可直接部署
大语言模型服务需求激增带来高昂部署与计算成本。模型路由通过根据模型和查询特征将请求分配至最优模型，提供成本效益解决方案。然而现有方法多聚焦离线场景，难以适应高查询量、受限令牌预算的在线环境。本文提出首个无需训练的在线路由算法。该算法利用近似最近邻搜索高效估算查询特征，并对少量初始查询进行一次性优化，学习出指导后续路由的策略。理论证明在自然假设下，算法具有1−o(1)的竞争比；在3个基准数据集、8个基线上的实验验证显示，平均性能提升3.55倍，成本效率提高1.85倍，吞吐量接近4.25倍。代码已公开于https://github.com/fzwark/PORT。
原文摘要 · Abstract (English)
Increasing demand for Large Language Models (LLMs) services imposes substantial deployment and computation costs on providers. LLM routing offers a cost-efficient solution by directing queries to the optimal LLM based on model and query features. However, existing works primarily focus on offline scenarios and struggle to adapt to online settings with high query volume and constrained token budgets. In this work, we introduce the first training-free algorithm for online routing scenarios. Our algorithm leverages approximate nearest neighbor search to efficiently estimate query features and performs a one-time optimization over a small set of initial queries to learn a routing strategy that guides future routing. We provide theoretical guarantees demonstrating that our algorithm achieves a competitive ratio of $1 - o(1)$ under natural assumptions, which is further validated by extensive experiments across 3 benchmark datasets and 8 baselines, showing an average improvement of 3.55$\times$ in overall performance, 1.85$\times$ in cost efficiency, and nearly 4.25$\times$ in throughput. Our code is available at https://github.com/fzwark/PORT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。