提出最优成本路由算法，让模型选择更便宜且够用。
CARROT: A Cost Aware Rate Optimal Router
- 基于成本与准确率预估，实现理论最优的路由决策
- 在多个数据集上比现有方法节省20%以上推理成本
- 适合追求低成本推理的AI系统部署者
随着大语言模型（LLMs）数量快速增长，研究者开始关注模型路由问题，即把查询导向成本最低但响应质量仍可接受的模型。本文通过极小极大分析建立了路由问题的下界，并发现仅需预测每个问题的成本与准确率，简单路由器即可达到最优。受此启发，我们提出CARROT——一种兼顾成本与性能估计的路由机制。同时构建了SPROUT数据集，覆盖广泛查询与最新SOTA LLMs，用于支持路由评估。基于SPROUT、Routerbench和open-LLM-leaderboard-v2等基准，实证验证了CARROT在多种场景下的优越性。
原文摘要 · Abstract (English)
With the rapid growth in the number of Large Language Models (LLMs), there has been a recent interest in LLM routing, or directing queries to the cheapest LLM that can deliver a suitable response. We conduct a minimax analysis of the routing problem, providing a lower bound and finding that a simple router that predicts both cost and accuracy for each question can be minimax optimal. Inspired by this, we introduce CARROT, a Cost AwaRe Rate Optimal rouTer that selects a model based on estimates of the models' cost and performance. Alongside CARROT, we also introduce the Smart Price-aware ROUTing (SPROUT) dataset to facilitate routing on a wide spectrum of queries with the latest state-of-the-art LLMs. Using SPROUT and prior benchmarks such as Routerbench and open-LLM-leaderboard-v2 we empirically validate CARROT's performance against several alternative routers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。