用分层路由+质量评估，让大模型推理又快又便宜
Cluster, Route, Escalate: Cascaded Framework for Cost-Aware LLM Serving
- 先聚类查询，选最省钱的模型处理
- 只在低质量时升级到强模型，节省97%-99%成本
- 无需重调参，自动适应新模型或任务变化
大型语言模型在生产环境中的高效部署需在准确率与成本间权衡。当前常采用单一模型，导致简单问题浪费资源，复杂问题又不够精准。为此，我们提出两阶段级联方案：第一阶段对输入查询进行聚类，并为每类分配最经济的模型，路由预算由可解释的超参数设定，离线调优；第二阶段引入质量评估（QE）级联机制，当第一阶段输出被判定为低质量时，查询将被升级至更强模型。该机制确保仅难例或低置信度案例进入昂贵模型。在测试数据集上，系统在保持最强模型97%-99%准确率的同时显著降低每输出词元时间（TPOT），仅需任务正确性标签，且无需人工重新配置即可适应模型池变化。
原文摘要 · Abstract (English)
Efficient deployment of large language models (LLMs) in production forces a trade-off between accuracy and cost. Operators often default to a single model that is either expensive for easy queries or insufficient for hard ones. To address this challenge, we propose a two-stage cascaded solution. Stage 1 clusters incoming queries and assigns each cluster to its most cost-effective model. The cost budget for this routing process is set by an interpretable hyperparameter, tuned offline. Stage 2 adds a quality estimation (QE) cascade; when an output from Stage 1 is judged low-quality, the query is escalated to a stronger model. This ensures only hard or low-confidence cases reach the expensive models. On the test datasets, the cascaded system retains 97-99% of the strongest model's accuracy while reducing Time Per Output Token (TPOT). It requires only task-correctness labels and adapts to changes in the model pool without manual reconfiguration.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。