将预训练模型转为专家混合架构，大幅降低语音识别训练成本。
UME: Upcycling Mixture-of-Experts for Scalable and Efficient Automatic Speech Recognition
- 用预训练权重重构前馈层为专家混合层，快速搭建大模型基础
- 在170小时中英文数据上实现11.9%相对错误率下降，延迟不变
- 训练时间减少86.7%，适合资源受限场景下的高效模型扩展
近年来，模型规模的扩大显著提升了自动语音识别（ASR）性能。然而，从头训练大型ASR模型仍代价高昂。为此，我们提出UME，一种将预训练密集型ASR检查点高效转换为更大规模混合专家（MoE）架构的新方法。首先，将前馈网络转化为MoE层，并复用预训练权重，为扩展模型建立坚实基础，显著缩短优化时间。随后采用层冻结与专家平衡策略继续训练，进一步提升性能。在包含170k小时普通话与英语数据的混合数据集上的实验表明：1）相比预训练基线，相对错误率降低11.9%，且延迟相当；2）训练时间最多减少86.7%，精度优于同规模从零训练的模型。
原文摘要 · Abstract (English)
Recent advancements in scaling up models have significantly improved performance in Automatic Speech Recognition (ASR) tasks. However, training large ASR models from scratch remains costly. To address this issue, we introduce UME, a novel method that efficiently Upcycles pretrained dense ASR checkpoints into larger Mixture-of-Experts (MoE) architectures. Initially, feed-forward networks are converted into MoE layers. By reusing the pretrained weights, we establish a robust foundation for the expanded model, significantly reducing optimization time. Then, layer freezing and expert balancing strategies are employed to continue training the model, further enhancing performance. Experiments on a mixture of 170k-hour Mandarin and English datasets show that UME: 1) surpasses the pretrained baseline by a margin of 11.9% relative error rate reduction while maintaining comparable latency; 2) reduces training time by up to 86.7% and achieves superior accuracy compared to training models of the same size from scratch.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。