通过逐步引入难样本,让模型学会更高效地理解语音。
EH-MAM: Easy-to-Hard Masked Acoustic Modeling for Self-Supervised Speech Representation Learning
- 根据帧级重建损失自动识别难处理区域进行掩码
- 在低资源语音识别任务上提升5%-10%性能
- 适合追求自监督语音表征的开发者和研究者
本文提出EH-MAM(Easy-to-Hard Masked Acoustic Modeling),一种新型自监督学习方法用于语音表征学习。与以往随机掩码策略不同,该方法采用可选且自适应的掩码机制:训练过程中逐步引入更难的语音片段供模型重建。其核心思想是利用每个帧的重建损失作为判断该帧难易程度的自然信号。通过一个教师模型先预测帧级损失,并据此决定哪些帧应被掩码。这种“主动制造难题”的方式使模型在识别并解决困难帧的同时,学习到更有效的语音表征,从而获得更全面的语音理解能力。定量结果显示,EH-MAM在多个低资源语音识别及SUPERB基准测试中均优于当前主流基线方法,性能提升达5%-10%。进一步分析表明,EH-MAM所掩码的区域能有效捕捉跨帧有用上下文信息。
原文摘要 · Abstract (English)
In this paper, we present EH-MAM (Easy-to-Hard adaptive Masked Acoustic Modeling), a novel self-supervised learning approach for speech representation learning. In contrast to the prior methods that use random masking schemes for Masked Acoustic Modeling (MAM), we introduce a novel selective and adaptive masking strategy. Specifically, during SSL training, we progressively introduce harder regions to the model for reconstruction. Our approach automatically selects hard regions and is built on the observation that the reconstruction loss of individual frames in MAM can provide natural signals to judge the difficulty of solving the MAM pre-text task for that frame. To identify these hard regions, we employ a teacher model that first predicts the frame-wise losses and then decides which frames to mask. By learning to create challenging problems, such as identifying harder frames and solving them simultaneously, the model is able to learn more effective representations and thereby acquire a more comprehensive understanding of the speech. Quantitatively, EH-MAM outperforms several state-of-the-art baselines across various low-resource speech recognition and SUPERB benchmarks by 5%-10%. Additionally, we conduct a thorough analysis to show that the regions masked by EH-MAM effectively capture useful context across speech frames.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。