动态调度多模型，让大模型回答更快更省钱。
MixLLM: Dynamic Routing in Mixed Large Language Models
- 用查询特征+轻量预测模型，实时判断哪个模型最合适。
- 在延迟约束下，达到GPT-4 97.25%质量，成本仅24.18%。
- 支持持续学习，能自动适应新模型和用户反馈。
大语言模型（LLM）展现通用智能潜力，但响应延迟高、成本昂贵。针对多种各有优劣的混合模型，通过动态路由为每个查询选择最合适的模型，可在保证质量的同时降低耗时与成本。现有挑战包括：质量、成本与延迟间的动态权衡；部署系统中持续学习能力；以及随时间变化的模型候选集（如新增或移除模型）。为此，我们提出MixLLM，一种基于上下文-贝叶斯带的动态路由系统。首先利用查询标签增强查询嵌入；其次设计轻量级预测模型，估算各模型在特定查询下的响应质量与成本；再通过元决策器权衡三者，选择最优分配方案；最后系统支持持续训练，可随查询变化与用户反馈自适应演进。大量实验表明，MixLLM在响应质量、成本与延迟间取得最佳平衡，在时间约束下实现97.25% GPT-4质量，仅需24.18%成本。
原文摘要 · Abstract (English)
Large Language Models (LLMs) exhibit potential artificial generic intelligence recently, however, their usage is costly with high response latency. Given mixed LLMs with their own strengths and weaknesses, LLM routing aims to identify the most suitable model for each query in the stream to maximize response quality and minimize cost and latency. However, the challenges involve: (1) dynamic trade-offs among quality, cost, and latency; (2) enabling continual learning in deployed systems; and (3) navigating a varying (e.g., new LLM addition or old LLM removal) set of LLM candidates over time. To bridge these gaps, we develop MixLLM, a dynamic contextual-bandit-based routing system for query-LLM assignment. Specifically, we first leverage query tags to enhance query embeddings for the routing task. Next, we design lightweight prediction models to estimate the response qualities and costs of queries over LLMs. We then devise a meta-decision maker to choose the query-LLM assignments to best tradeoff response quality, cost, and latency. Finally, the system benefits from continual training, allowing it to adapt to evolving queries and user feedback over time. Our extensive experiments show that MixLLM achieves the best trade-offs in response quality, cost, and latency (97.25% of GPT-4's quality at 24.18% of the cost under the time constraint).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。