用能力指令动态匹配最优大模型，提升整体性能。
Capability Instruction Tuning: A New Paradigm for Dynamic LLM Routing
- 构建能力指令评估模型表现，实现精准路由
- 新框架Model-SAT无需候选模型推理，实时匹配最佳模型
- 新模型可快速完成50个任务的适性测试，适合部署新模型
大型语言模型（LLM）在超过1000亿参数时展现出类人指令遵循能力。一些小型、资源友好的模型组合可覆盖大多数大型模型擅长的任务。本文探索如何为每条指令动态路由最佳模型以提升整体性能。提出一种新范式：通过模型能力表示、用户指令和性能询问提示构建能力指令来评估模型表现。设计端到端框架Model-SAT，基于不同模型在任务中的优劣生成正负样本，并引入模型能力编码器将模型表示扩展至轻量级模型。实验表明，Model-SAT能理解候选模型的性能维度，输出其处理各类指令的能力概率。部署阶段，新模型可在50个任务上、每个任务20个样本下快速推断自身适性测试结果。该方法在无候选推理情况下达到当前最优模型路由效果，适用于真实世界中新模型发布场景。代码已开源。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have demonstrated human-like instruction-following abilities, particularly those exceeding 100 billion parameters. The combined capability of some smaller, resource-friendly LLMs can address most of the instructions that larger LLMs excel at. In this work, we explore how to route the best-performing LLM for each instruction to achieve better overall performance. We develop a new paradigm, constructing capability instructions with model capability representation, user instruction, and performance inquiry prompts to assess the performance. To learn from capability instructions, we introduce a new end-to-end framework called Model Selection with Aptitude Test (Model-SAT), which generates positive and negative samples based on what different models perform well or struggle with. Model-SAT uses a model capability encoder that extends its model representation to a lightweight LLM. Our experiments show that Model-SAT understands the performance dimensions of candidate models and provides the probabilities of their capability to handle various instructions. Additionally, during deployment, a new model can quickly infer its aptitude test results across 50 tasks, each with 20 shots. Model-SAT performs state-of-the-art model routing without candidate inference and in real-world new model-released scenarios. The code is available at https://github.com/Now-Join-Us/CIT-LLM-Routing
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。