用模型合并实现近乎免费的测试时训练，让专家模型规模扩大10倍。
Local Mixtures of Experts: Essentially Free Test-Time Training via Model Merging
- 通过模型合并，在测试时动态组合大量专家，避免逐次微调开销。
- 10亿参数模型下，测试速度比直接测试时训练快100倍以上。
- 适合需要高适应性但算力受限的场景，如边缘部署或个性化服务。
混合专家（MoE）模型是提升模型容量而不增加推理成本的有前途方法，也是许多顶尖语言模型的核心组件。然而，当前的MoE模型通常仅使用少量专家，因训练和推理成本过高。我们提出测试时模型合并（TTMM），将MoE范式扩展至多一个数量级的专家，并利用模型合并避免几乎所有的测试时开销。我们发现TTMM是测试时训练（TTT）的一种近似，即为每个预测任务（提示）微调一个专家模型。尽管TTT最近被证明能显著提升语言模型性能，但其计算成本高昂。我们发现TTMM的性能随专家数量增加而提升，接近TTT表现。此外，使用10亿参数的基础模型，TTMM在测试时比TTT快超过100倍，因为将TTT的成本分摊到了训练阶段。因此，TTMM提供了一种高效低成本的测试时训练扩展方案。
原文摘要 · Abstract (English)
Mixture of expert (MoE) models are a promising approach to increasing model capacity without increasing inference cost, and are core components of many state-of-the-art language models. However, current MoE models typically use only few experts due to prohibitive training and inference cost. We propose Test-Time Model Merging (TTMM) which scales the MoE paradigm to an order of magnitude more experts and uses model merging to avoid almost any test-time overhead. We show that TTMM is an approximation of test-time training (TTT), which fine-tunes an expert model for each prediction task, i.e., prompt. TTT has recently been shown to significantly improve language models, but is computationally expensive. We find that performance of TTMM improves with more experts and approaches the performance of TTT. Moreover, we find that with a 1B parameter base model, TTMM is more than 100x faster than TTT at test-time by amortizing the cost of TTT at train-time. Thus, TTMM offers a promising cost-effective approach to scale test-time training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。