用双对比学习构建查询路由，让多个大模型协作更高效。
RouterDC: Query-Based Router by Dual Contrastive Learning for Assembling Large Language Models
- 基于查询的路由模型，通过双对比学习训练
- 在分布内/外任务上均比单个顶尖模型高2.76%和1.90%
- 适合多模型集成场景，尤其擅长处理性能相近的模型
近期研究显示，整合多个现成的大语言模型（LLMs）可发挥其互补优势。为此，路由机制是一种有前景的方法，即学习一个路由器为每个查询选择最合适的LLM。然而，当多个LLM对同一查询表现良好时，现有路由模型效果不佳。为此，本文提出一种基于查询的双对比学习路由方法（RouterDC）。该模型包含编码器与LLM嵌入，并设计了两种对比学习损失来训练。实验结果表明，RouterDC在组装多个LLM方面有效，在分布内任务上较最优单个模型提升2.76%，在分布外任务上提升1.90%，显著优于现有路由方法。代码已开源。
原文摘要 · Abstract (English)
Recent works show that assembling multiple off-the-shelf large language models (LLMs) can harness their complementary abilities. To achieve this, routing is a promising method, which learns a router to select the most suitable LLM for each query. However, existing routing models are ineffective when multiple LLMs perform well for a query. To address this problem, in this paper, we propose a method called query-based Router by Dual Contrastive learning (RouterDC). The RouterDC model consists of an encoder and LLM embeddings, and we propose two contrastive learning losses to train the RouterDC model. Experimental results show that RouterDC is effective in assembling LLMs and largely outperforms individual top-performing LLMs as well as existing routing methods on both in-distribution (+2.76\%) and out-of-distribution (+1.90\%) tasks. Source code is available at https://github.com/shuhao02/RouterDC.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。