让大模型推理更省资源，能自动选最合适的模型。
Universal Model Routing for Efficient LLM Inference
- 用代表性问题生成模型特征向量，动态匹配最优模型。
- 在30多个未见过的模型中测试，显著降低推理成本。
- 适合需要快速部署新模型的AI系统开发者。
模型路由是一种降低大语言模型（LLMs）推理成本的简单技术，即维护一个候选模型池，并学习将每个输入提示路由到最小可行的模型。现有方法专注于为固定模型池学习路由器。本文研究动态路由问题，即测试时可获得之前未见过的新模型。我们提出UniRoute，一种新方法，将每个LLM表示为基于一组代表性提示预测所得的特征向量。基于此，我们设计了两种有效实现：基于聚类的路由和基于学习的聚类映射。我们证明这些方法是理论上最优路由规则的估计，并通过超额风险界量化其误差。在多个公开基准上的实验表明，UniRoute在路由超过30个未见过的LLM时表现优异。
原文摘要 · Abstract (English)
Model routing is a simple technique for reducing the inference cost of large language models (LLMs), wherein one maintains a pool of candidate LLMs, and learns to route each prompt to the smallest feasible LLM. Existing works focus on learning a router for a fixed pool of LLMs. In this paper, we consider the problem of dynamic routing, where new, previously unobserved LLMs are available at test time. We propose UniRoute, a new approach to this problem that relies on representing each LLM as a feature vector, derived based on predictions on a set of representative prompts. Based on this, we detail two effective instantiations of UniRoute, relying on cluster-based routing and a learned cluster map respectively. We show that these are estimates of a theoretically optimal routing rule, and quantify their errors via an excess risk bound. Experiments on a range of public benchmarks show the effectiveness of UniRoute in routing amongst more than 30 unseen LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。