提出k2SSL框架，用Zipformer提升语音自监督学习效率与性能
k2SSL: A Faster and Better Framework for Self-Supervised Speech Representation Learning
- 采用优化的Zipformer结构替代传统Transformer，提升训练效率
- 在LibriSpeech上相较HuBERT Base降低34.8%识别错误率，训练速度提升3.5倍
- 适合追求高效语音预训练的ASR研究者与工业部署场景
自监督学习（SSL）在语音任务中取得显著进展。尽管Transformer和Conformer已成为主流架构，但擅长自动语音识别（ASR）的Zipformer编码器尚未在SSL中充分探索。同时，现有框架如fairseq在数据处理方面存在效率瓶颈，难以应对日益增长的训练数据量。为此，我们提出k2SSL——一个开源的更快、更节省内存且表现更优的自监督语音表征学习框架，专注于下游ASR任务。优化后的HuBERT及提出的基于Zipformer的SSL系统在训练时间和内存占用上均有显著降低。在LibriSpeech上的实验表明，Zipformer Base在微调后相比HuBERT Base相对减少34.8%的词错误率（WER），GPU小时预训练速度提升3.5倍。当扩展至60,000小时的LibriLight数据时，Zipformer Large仅需5/8的预训练步数即可达到与HuBERT Large相当的性能。
原文摘要 · Abstract (English)
Self-supervised learning (SSL) has achieved great success in speech-related tasks. While Transformer and Conformer architectures have dominated SSL backbones, encoders like Zipformer, which excel in automatic speech recognition (ASR), remain unexplored in SSL. Concurrently, inefficiencies in data processing within existing SSL training frameworks, such as fairseq, pose challenges in managing the growing volumes of training data. To address these issues, we propose k2SSL, an open-source framework that offers faster, more memory-efficient, and better-performing self-supervised speech representation learning, focusing on downstream ASR tasks. The optimized HuBERT and proposed Zipformer-based SSL systems exhibit substantial reductions in both training time and memory usage during SSL training. Experiments on LibriSpeech demonstrate that Zipformer Base significantly outperforms HuBERT and WavLM, achieving up to a 34.8% relative WER reduction compared to HuBERT Base after fine-tuning, along with a 3.5x pre-training speedup in GPU hours. When scaled to 60k hours of LibriLight data, Zipformer Large exhibits remarkable efficiency, matching HuBERT Large's performance while requiring only 5/8 pre-training steps.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。