用两阶段训练提升多语言语音识别离散令牌表现
Multilingual Speech Recognition Using Discrete Tokens with a Two-step Training Strategy
- 提出两阶段训练策略优化预训练模型的离散令牌表示
- 在ML-SUPERB上使XLS-R的词错误率降低44%（相对）
- 适用于追求高效低存储的多语言语音识别场景
预训练模型，尤其是自监督学习（SSL）模型，在自动语音识别（ASR）任务中表现优异。尽管多数应用聚焦于使用连续表示作为下游任务特征，但离散单元因其更低的存储需求和更广的应用范围，近年来受到越来越多关注。在多语言ASR任务中，模型不同层的表示对不同语言的贡献各异，使得离散单元建模统一化变得复杂。本文提出一种两阶段训练策略，以提升预训练模型的离散令牌性能，并缩小其与连续表示性能之间的差距。我们在XLS-R模型上验证该方法，采用Interspeech2024语音处理离散语音单元挑战赛的设置。实验结果表明，该方法在ML-SUPERB数据集上实现显著提升，使XLS-R的词错误率（CER）相对降低44%。这一表现超越了此前由WavLM模型设定的26%相对降低基准。此外，该方法在排行榜上获得单系统第一名。
原文摘要 · Abstract (English)
Pre-trained models, especially self-supervised learning (SSL) models, have demonstrated impressive results in automatic speech recognition (ASR) task. While most applications of SSL models focus on leveraging continuous representations as features for training downstream tasks, the utilization of discrete units has gained increasing attention in recent years owing to its lower storage requirements and broader range of applications. In multilingual ASR tasks, representations at different layers of the model contribute differently to various languages, complicating the unification of discrete unit modeling. In this paper, we propose a two-stage training strategy to improve the discrete token performance of pre-trained models and narrow the gap with continuous representation performance. We validate our method on the XLS-R model following the settings of Interspeech2024 Speech Processing Using Discrete Speech Unit Challenge. Our method demonstrates a significant improvement on the ML-SUPERB dataset, achieving a 44% relative reduction on CER for the XLS-R model. This surpasses the previous baseline set by the WavLM model, which achieves a 26% relative reduction on CER. Furthermore, our method achieves the first place among all the single-system results on the leaderboard.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。