用评测数据训练专家协作系统，让多个模型联手完成任务。
Bench-CoE: a Framework for Collaboration of Experts from Benchmark
- 基于评测数据训练路由模型，智能分配任务给最合适专家
- 在语言与多模态任务上均超越单一模型表现
- 适合需要多模型协同的复杂任务场景
大型语言模型（LLMs）是推动智能系统处理多任务的关键技术。为应对各类任务需求，越来越多具备不同能力的LLM驱动专家被开发，并配套相应的基准测试集用于评估性能。本文提出Bench-CoE框架，通过有效利用基准评测结果实现专家间的协作（CoE），以在多种任务中取得最优表现。该框架包含一组专家模型、一个用于将任务分配给对应专家的路由模块，以及一个用于训练路由模块的基准数据集。此外，我们基于该框架提出了查询级和主题级两种方法，并分析了其优劣。最后，在语言与多模态任务上，针对不同数据分布进行了系列实验，验证了所提Bench-CoE在整体性能上优于任意单一模型。我们希望该方法能成为该领域进一步研究的基准。代码已开源。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are key technologies driving intelligent systems to handle multiple tasks. To meet the demands of various tasks, an increasing number of LLMs-driven experts with diverse capabilities have been developed, accompanied by corresponding benchmarks to evaluate their performance. This paper proposes the Bench-CoE framework, which enables Collaboration of Experts (CoE) by effectively leveraging benchmark evaluations to achieve optimal performance across various tasks. Bench-CoE includes a set of expert models, a router for assigning tasks to corresponding experts, and a benchmark dataset for training the router. Moreover, we formulate Query-Level and Subject-Level approaches based on our framework, and analyze the merits and drawbacks of these two approaches. Finally, we conduct a series of experiments with vary data distributions on both language and multimodal tasks to validate that our proposed Bench-CoE outperforms any single model in terms of overall performance. We hope this method serves as a baseline for further research in this area. The code is available at \url{https://github.com/ZhangXJ199/Bench-CoE}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。