用图匹配优化传输,让语音和语言模型更好对齐。
Cross-modal Knowledge Transfer Learning as Graph Matching Based on Optimal Transport for ASR
- 将语音与语言特征建模为带关系的图,用最优传输对齐
- 在中文语音识别上提升性能,超越现有最佳方法
- 适合做语音识别中跨模态知识迁移的研究者
将预训练语言模型(PLM)中的语言知识迁移到声学特征学习中,已被证明能有效提升端到端语音识别(E2E-ASR)性能。然而,由于语义与声学模态间的固有差异,二者表示对齐仍具挑战。最优传输(OT)通过最小化语言与声学特征分布间的Wasserstein距离(WD)展现潜力。但已有方法忽略特征间的结构关系,将特征向量视为无序集合。为此,本文提出图匹配最优传输(GM-OT),将语言与声学序列建模为结构图:节点代表特征嵌入,边捕获时序与顺序关系。GM-OT同时最小化节点间的WD与边间的Gromov-Wasserstein距离(GWD),形成融合的Gromov-Wasserstein距离(FGWD)公式。该方法实现更优的结构化对齐,相比现有基于OT的方法更具知识迁移效率。理论分析表明,先前基于OT的语言知识迁移方法可视为本框架的特例。我们在基于CTC的端到端中文语音识别系统上评估了GM-OT,使用PLM进行知识迁移。实验结果表明,该方法显著优于当前最佳模型,验证了其有效性。
原文摘要 · Abstract (English)
Transferring linguistic knowledge from a pretrained language model (PLM) to acoustic feature learning has proven effective in enhancing end-to-end automatic speech recognition (E2E-ASR). However, aligning representations between linguistic and acoustic modalities remains a challenge due to inherent modality gaps. Optimal transport (OT) has shown promise in mitigating these gaps by minimizing the Wasserstein distance (WD) between linguistic and acoustic feature distributions. However, previous OT-based methods overlook structural relationships, treating feature vectors as unordered sets. To address this, we propose Graph Matching Optimal Transport (GM-OT), which models linguistic and acoustic sequences as structured graphs. Nodes represent feature embeddings, while edges capture temporal and sequential relationships. GM-OT minimizes both WD (between nodes) and Gromov-Wasserstein distance (GWD) (between edges), leading to a fused Gromov-Wasserstein distance (FGWD) formulation. This enables structured alignment and more efficient knowledge transfer compared to existing OT-based approaches. Theoretical analysis further shows that prior OT-based methods in linguistic knowledge transfer can be viewed as a special case within our GM-OT framework. We evaluate GM-OT on Mandarin ASR using a CTC-based E2E-ASR system with a PLM for knowledge transfer. Experimental results demonstrate significant performance gains over state-of-the-art models, validating the effectiveness of our approach.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。