针对多源语音数据，提出局部约束预训练方法提升模型适应性。
Heterogeneous Self-Supervised Acoustic Pre-Training with Local Constraints
- 通过局部约束优化各数据源，避免全局平均损失的干扰
- 在多领域多语言数据上显著提升下游任务性能
- 适合需要跨域跨语言迁移的语音识别场景
无监督语音预训练广泛应用于自动语音识别。本文提出一种处理异构数据的新方法：不将所有数据混合并最小化平均全局损失，而是施加额外的局部约束，确保模型在从当前模型初始化的条件下，经K步梯度下降后能优化到每个异构数据源的局部最优。该问题被建模为双层优化，并采用一阶近似求解。我们讨论了其与模型无关元学习的关联。在多领域、多语言数据集上的实验表明，该方法可显著提升自监督预训练模型在下游监督微调任务中的适应能力。
原文摘要 · Abstract (English)
Self-supervised pre-training using unlabeled data is widely used in automatic speech recognition. In this paper, we propose a new self-supervised pre-training approach to dealing with heterogeneous data. Instead of mixing all the data and minimizing the averaged global loss in the conventional way, we impose additional local constraints to ensure that the model optimizes each source of heterogeneous data to its local optimum after $K$-step gradient descent initialized from the model. We formulate this as a bilevel optimization problem, and use the first-order approximation method to solve the problem. We discuss its connection to model-agnostic meta learning. Experiments are carried out on self-supervised pre-training using multi-domain and multilingual datasets, demonstrating that the proposed approach can significantly improve the adaptivity of the self-supervised pre-trained model for the downstream supervised fine-tuning tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。