提出批量路由框架，在资源受限下提升大模型推理的稳定性与效率。
Robust Batch-Level Query Routing for Large Language Models under Cost and Capacity Constraints
- 按批次统一调度，兼顾成本与显存容量限制
- 对抗性批量下准确率提升最多14%，比逐查询调度高24%
- 支持性能不确定性建模，适合多模型部署场景
我们研究在成本、GPU资源和并发量约束下，将查询路由至大语言模型（LLMs）的问题。以往的逐查询路由方法难以控制批量层面的成本，尤其在非均匀或对抗性批量情况下表现不佳。为此，我们提出一种面向批次、资源感知的路由框架，联合优化每一批次的模型分配，同时满足成本和模型容量限制。进一步引入鲁棒变体，考虑预测的LLM性能不确定性，并设计离线实例分配流程，以平衡多模型间的质量与吞吐。在两个多任务LLM基准上的实验表明，该方法在不同性能估计器下，鲁棒性使准确率提升1%-14%；在对抗性批量下，批量路由相比逐查询方法最高提升24%；优化实例分配相较非优化方案额外带来最高3%的增益，所有结果均严格遵守成本与GPU资源约束。
原文摘要 · Abstract (English)
We study the problem of routing queries to large language models (LLMs) under cost, GPU resources, and concurrency constraints. Prior per-query routing methods often fail to control batch-level cost, especially under non-uniform or adversarial batching. To address this, we propose a batch-level, resource-aware routing framework that jointly optimizes model assignment for each batch while respecting cost and model capacity limits. We further introduce a robust variant that accounts for uncertainty in predicted LLM performance, along with an offline instance allocation procedure that balances quality and throughput across multiple models. Experiments on two multi-task LLM benchmarks show that robustness improves accuracy by 1-14% over non-robust counterparts (depending on the performance estimator), batch-level routing outperforms per-query methods by up to 24% under adversarial batching, and optimized instance allocation yields additional gains of up to 3% compared to a non-optimized allocation, all while strictly controlling cost and GPU resource constraints.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。