用合成数据提升语音识别个性化,同时不丢弃通用知识。
Knowledge-Decoupled Functionally Invariant Path with Synthetic Personal Data for Personalized ASR
- 分离存储通用与个人知识,分步通过不变路径学习
- 合成数据下字符错误率降低29.38%,通用性能基本不变
- 适合需要高个性化又怕遗忘通用能力的语音系统
使用大规模合成个人数据微调通用语音识别模型可提升个性化效果,但会面临在适应合成数据时遗忘真实知识、在适应个人数据时遗忘通用知识的问题。考虑到功能不变路径(FIP)框架能实现模型适配同时保留先验知识,本文将FIP引入合成数据增强的个性化语音识别模型。然而,当同时在三种数据上训练时,模型仍难以平衡合成、个人及通用知识的学习。为此,本文在FIP中引入门控参数隔离策略,提出知识解耦的功能不变路径(KDFIP)框架,将通用与个人知识分别存于独立模块,并依次应用FIP进行训练。具体而言,个性化模块用于适应合成与真实个人数据,通用模块用于通用数据,两者均沿个性化不变路径更新,输出通过门控机制动态融合。在合成数据增强下,KDFIP在目标说话人上实现29.38%的相对字符错误率下降,且保持与未适配基线相当的泛化性能。
原文摘要 · Abstract (English)
Fine-tuning generic ASR models with large-scale synthetic personal data can enhance the personalization of ASR models, but it introduces challenges in adapting to synthetic personal data without forgetting real knowledge, and in adapting to personal data without forgetting generic knowledge. Considering that the functionally invariant path (FIP) framework enables model adaptation while preserving prior knowledge, in this letter, we introduce FIP into synthetic-data-augmented personalized ASR models. However, the model still struggles to balance the learning of synthetic, personalized, and generic knowledge when applying FIP to train the model on all three types of data simultaneously. To decouple this learning process and further address the above two challenges, we integrate a gated parameter-isolation strategy into FIP and propose a knowledge-decoupled functionally invariant path (KDFIP) framework, which stores generic and personalized knowledge in separate modules and applies FIP to them sequentially. Specifically, KDFIP adapts the personalized module to synthetic and real personal data and the generic module to generic data. Both modules are updated along personalization-invariant paths, and their outputs are dynamically fused through a gating mechanism. With augmented synthetic data, KDFIP achieves a 29.38% relative character error rate reduction on target speakers and maintains comparable generalization performance to the unadapted ASR baseline.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。