动态部署和路由大模型，高效控制成本与吞吐。
Near-Optimal Online Deployment and Routing for Streaming LLMs
- 分阶段部署模型，按预算和性能动态选择最优组合。
- 实测在严苛预算下逼近理想解，支持多种负载场景。
- 适合需要持续更新模型的实时服务系统使用。
新大模型快速迭代，旧模型迅速过时，服务商需在严格并发限制和每查询成本预算下管理流式模型库存。本文将此问题建模为在线决策问题，耦合阶段式部署（固定维护窗口）与查询级路由。提出StageRoute算法：(i) 基于奖励上界与成本下界的乐观估计，选择最多$M_{ ext{max}}$个模型进入下一阶段；(ii) 对每个查询求解一个受预算与吞吐约束的多臂赌博机子问题以决定路由。理论证明该算法具有$ ilde{ ext{O}}(T^{2/3})$的后悔率，并给出匹配下界，实现近似最优。实证验证其在多种工作负载下可紧追理想基准，在严苛预算下表现优异。
原文摘要 · Abstract (English)
The rapid pace at which new large language models (LLMs) appear, and older ones become obsolete, forces providers to manage a streaming inventory under a strict concurrency cap and per-query cost budgets. We cast this as an online decision problem that couples stage-wise deployment (at fixed maintenance windows) with per-query routing among live models. We introduce StageRoute, a hierarchical algorithm that (i) optimistically selects up to $M_{\max}$ models for the next stage using reward upper-confidence and cost lower-confidence bounds, and (ii) routes each incoming query by solving a budget- and throughput-constrained bandit subproblem over the deployed set. We prove a regret of $\tilde{\mathcal{O}}(T^{2/3})$ with a matching lower bound, establishing near-optimality, and validate the theory empirically: StageRoute tracks a strong oracle under tight budgets across diverse workloads.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。