同时优化模型架构与参数配置，提升大模型任务适配能力
HAPS: Hierarchical LLM Routing with Joint Architecture and Parameter Search
- 分层路由：先选架构，再优化参数
- 联合搜索使性能超越现有基线
- 适合需要高效调优大模型的开发者
大型语言模型（LLM）路由旨在利用不同LLM在多样化任务中的专长。然而，现有方法通常只关注模型架构的选择，忽略了对参数设置的考量，而后者对任务表现至关重要。本文提出HAPS，一种分层的LLM路由框架，能够联合搜索模型架构与参数。具体而言，高层路由器从候选架构中进行选择，随后由低层路由器针对选定架构搜索最优参数。我们设计了一个参数生成网络，在两个路由器间共享参数，以相互增强其能力。在训练过程中，引入奖励增强目标函数以有效优化整体框架。在两个常用基准上的实验表明，HAPS持续优于强基线方法。代码已开源：https://github.com/zihangtian/HAPS。
原文摘要 · Abstract (English)
Large language model (LLM) routing aims to exploit the specialized strengths of different LLMs for diverse tasks. However, existing approaches typically focus on selecting LLM architectures while overlooking parameter settings, which are critical for task performance. In this paper, we introduce HAPS, a hierarchical LLM routing framework that jointly searches over model architectures and parameters. Specifically, we use a high-level router to select among candidate LLM architectures, and then search for the optimal parameters for the selected architectures based on a low-level router. We design a parameter generation network to share parameters between the two routers to mutually enhance their capabilities. In the training process, we design a reward-augmented objective to effectively optimize our framework. Experiments on two commonly used benchmarks show that HAPS consistently outperforms strong routing baselines. We have released our code at https://github.com/zihangtian/HAPS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。