根据问题难易动态选择模型和生成次数，大幅降本近60%。
BEST-Route: Adaptive LLM Routing with Test-Time Optimal Compute
- 按查询难度和质量阈值自适应选模型与采样数量
- 实测成本降低60%，性能损失小于1%
- 适合追求高性价比的LLM部署场景
大语言模型虽强大，但规模化部署成本高昂。查询路由通过动态分配不同成本与质量的模型来平衡开销与效果。以往方法仅生成一次响应，小模型单次输出常无法超越大模型，导致过度依赖昂贵模型，错失降本机会。我们发现，对小模型生成多份答案并择优，可显著提升质量且仍低于单次大模型输出成本。为此提出BEST-Route框架，根据查询难度与质量阈值动态决定模型选择及采样数量。在真实数据集上的实验表明，该方法可实现最高60%的成本降低，性能损失低于1%。
原文摘要 · Abstract (English)
Large language models (LLMs) are powerful tools but are often expensive to deploy at scale. LLM query routing mitigates this by dynamically assigning queries to models of varying cost and quality to obtain a desired trade-off. Prior query routing approaches generate only one response from the selected model and a single response from a small (inexpensive) model was often not good enough to beat a response from a large (expensive) model due to which they end up overusing the large model and missing out on potential cost savings. However, it is well known that for small models, generating multiple responses and selecting the best can enhance quality while remaining cheaper than a single large-model response. We leverage this idea to propose BEST-Route, a novel routing framework that chooses a model and the number of responses to sample from it based on query difficulty and the quality thresholds. Experiments on real-world datasets demonstrate that our method reduces costs by up to 60% with less than 1% performance drop.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。