arXiv:2501.13497cs.SDcs.CL2025-01被引 1

通过解耦量化提升多语言语音识别效果,让模型更专注区分语言和音素。

DQ-Data2vec: Decoupling Quantization for Multilingual Speech Recognition

  • 用指定聚类数的K-means量化器分离语言与音素特征,避免多层平均带来的信息耦合。
  • 在CommonVoice上,音素错误率降低9.51%,词错误率降低11.58%。
  • 适合需要精确区分多种语言或音素的多语言语音识别场景。

Data2vec是一种自监督学习方法,采用教师-学生架构通过掩码预测进行上下文表示学习,在单语语音识别中表现优异。已有研究发现,Data2vec的浅层捕捉说话人和语言信息,中层编码音素与词汇特征,深层负责重建。语言与音素特征对多语言语音识别至关重要。然而,Data2vec的掩码表示生成依赖多层平均,不可避免地耦合了这些特征。为此,本文提出基于解耦量化的Data2vec(DQ-Data2vec),包含Data2vec主干和两个改进的在线K-means量化器。核心思想是使用指定聚类数的K-means量化器,对掩码预测中的语言与音素信息进行解耦。具体而言,在语言量化中,因语言数量远少于其他无关特征(如说话人),将聚类数设为语言数量,显式分离浅层的语言相关信息。该策略同样应用于中层音素与词汇特征的解耦。在自监督场景下,于CommonVoice数据集上的实验表明,与Data2vec和UniData2vec相比,DQ-Data2vec在音素错误率(PER)上相对减少9.51%,词错误率(WER)减少11.58%。在弱监督场景下,结合语言标签和高资源语言文本标签时,相对减少分别为18.09%和1.55%。

原文摘要 · Abstract (English)

Data2vec is a self-supervised learning (SSL) approach that employs a teacher-student architecture for contextual representation learning via masked prediction, demonstrating remarkable performance in monolingual ASR. Previous studies have revealed that data2vec's shallow layers capture speaker and language information, middle layers encode phoneme and word features, while deep layers are responsible for reconstruction. Language and phoneme features are crucial for multilingual ASR. However, data2vec's masked representation generation relies on multi-layer averaging, inevitably coupling these features. To address this limitation, we propose a decoupling quantization based data2vec (DQ-Data2vec) for multilingual ASR, which includes a data2vec backbone and two improved online K-means quantizers. Our core idea is using the K-means quantizer with specified cluster numbers to decouple language and phoneme information for masked prediction. Specifically, in the language quantization, considering that the number of languages is significantly different from other irrelevant features (e.g., speakers), we assign the cluster number to match the number of languages, explicitly decoupling shallow layers' language-related information from irrelevant features. This strategy is also applied to decoupling middle layers' phoneme and word features. In a self-supervised scenario, experiments on the CommonVoice dataset demonstrate that DQ-Data2vec achieves a relative reduction of 9.51% in phoneme error rate (PER) and 11.58% in word error rate (WER) compared to data2vec and UniData2vec. Moreover, in a weakly-supervised scenario incorporating language labels and high-resource language text labels, the relative reduction is 18.09% and 1.55%, respectively.

多语言识别自监督学习解耦量化语音识别

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。