无需训练，动态组合专家模型提升任务表现
SpectR: Dynamically Composing LM Experts with Spectral Routing
- 按需动态选择不同层、不同词元的专家模型
- 相比现有方法路由准确率更高，跨领域任务表现更好
- 适合需要灵活调用多个专用模型的应用场景
训练通用大语言模型面临巨大挑战。随着从预训练模型微调出的专用专家模型日益丰富，为特定任务或领域提供了一种有前景的替代方案。在真实应用中有效利用这些已有专家模型，关键在于能否高效地选择或融合最适合当前任务的模型。本文提出SPECTR，一种在推理过程中每一步动态组合专家模型的方法。该方法无需额外训练，支持灵活的词元级与层级模型组合。实验表明，SPECTR在无需训练的情况下，显著提升了路由准确性，进一步增强了跨专家领域的任务性能。
原文摘要 · Abstract (English)
Training large, general-purpose language models poses significant challenges. The growing availability of specialized expert models, fine-tuned from pretrained models for specific tasks or domains, offers a promising alternative. Leveraging the potential of these existing expert models in real-world applications requires effective methods to select or merge the models best suited for a given task. This paper introduces SPECTR, an approach for dynamically composing expert models at each time step during inference. Notably, our method requires no additional training and enables flexible, token- and layer-wise model combinations. Our experimental results demonstrate that SPECTR improves routing accuracy over alternative training-free methods, increasing task performance across expert domains.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。