让软件自动优化大模型调用，省钱又省力
Real-time Adapting Routing (RAR): Improving Efficiency Through Continuous Learning in Software Powered by Layered Foundation Models
- 用实时学习动态调整请求分配，减少对高价模型依赖
- 减少50.2%对高成本模型的调用，响应质量仍保持90.5%以上
- 通过强模型生成提示，提升弱模型表现，适合部署优化场景
为平衡基础模型（如大语言模型）驱动软件的质量与推理开销，人们常训练路由模型将请求分发至不同规模和能力的模型。现有路由模型依赖精心标注数据学习最优决策，更新复杂，且忽略弱模型的潜力。本文提出实时自适应路由（RAR），通过持续学习动态调整路由，并利用引导式上下文学习增强弱模型能力，以降低对强模型的依赖。我们在MMLU基准的不同子集上评估该方法，结果表明，随着时间推移，该方法使需调用计算昂贵模型的请求减少50.2%，同时维持约90.5%的整体响应质量。此外，由强模型生成的引导具备域内泛化能力，显著优于独立弱模型的等效方案。
原文摘要 · Abstract (English)
To balance the quality and inference cost of a Foundation Model (FM, such as large language models (LLMs)) powered software, people often opt to train a routing model that routes requests to FMs with different sizes and capabilities. Existing routing models rely on learning the optimal routing decision from carefully curated data, require complex computations to be updated, and do not consider the potential evolution of weaker FMs. In this paper, we propose Real-time Adaptive Routing (RAR), an approach to continuously adapt FM routing decisions while using guided in-context learning to enhance the capabilities of weaker FM. The goal is to reduce reliance on stronger, more expensive FMs. We evaluate our approach on different subsets of the popular MMLU benchmark. Over time, our approach routes 50.2% fewer requests to computationally expensive models while maintaining around 90.5% of the general response quality. In addition, the guides generated from stronger models have shown intra-domain generalization and led to a better quality of responses compared to an equivalent approach with a standalone weaker FM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。