通过参数平均快速生成适配不同数据分布的专用模型
Soup-of-Experts: Pretraining Specialist Models via Parameters Averaging
- 用专家参数库线性组合生成模型，按需切换领域权重
- 仅需一次训练即可支持多种领域组合，速度极快
- 适合在模型大小受限时快速部署多个专用模型
机器学习模型通常在多种数据域混合上训练，不同领域权重会导致下游性能显著差异。本文提出 Soup-of-Experts 架构，可在测试时以极低计算成本、无需重训练，为任意领域权重组合实例化模型。该架构由一组专家参数构成，通过输入领域权重动态线性组合生成目标模型。训练时随机采样领域权重，实例化对应模型，并对单批次数据反向传播更新系数。实验表明，该方法能在多个语言建模任务中快速获得小型专用模型。在模型尺寸受限且需快速交付多套专用模型的场景下尤为适用。
原文摘要 · Abstract (English)
Machine learning models are routinely trained on a mixture of different data domains. Different domain weights yield very different downstream performances. We propose the Soup-of-Experts, a novel architecture that can instantiate a model at test time for any domain weights with minimal computational cost and without re-training the model. Our architecture consists of a bank of expert parameters, which are linearly combined to instantiate one model. We learn the linear combination coefficients as a function of the input domain weights. To train this architecture, we sample random domain weights, instantiate the corresponding model, and backprop through one batch of data sampled with these domain weights. We demonstrate how our approach obtains small specialized models on several language modeling tasks quickly. Soup-of-Experts are particularly appealing when one needs to ship many different specialist models quickly under a model size constraint.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。