通过自条件CTC与知识迁移，提升RNN-T模型的语音理解性能。
Improving Transducer-Based Spoken Language Understanding with Self-Conditioned CTC and Knowledge Transfer
- 用自条件CTC联合建模语音识别与语义理解，确保任务间依赖关系。
- 在SLU任务上显著优于仅优化语义理解的基线模型。
- 适配小模型实现接近Whisper大模型的性能，适合资源受限场景。
本文提出在基于RNN转录器（RNN-T）的端到端语音语言理解（SLU）中，引入联合自条件CTC语音识别（ASR）目标以提升性能。所提模型类似于可微分级联结构，依次执行语音识别与语义理解，并通过CTC自条件机制确保语义理解任务依赖于语音识别结果。该联合建模方式显著提升了SLU表现。进一步通过将声学嵌入与语义更丰富的BERT模型对齐，采用实体词袋预测层生成条件信号，用于指导RNN-T的语义解码。实验表明，该方法在多个强基线上取得显著提升，性能媲美大型模型Whisper，但参数量显著更少。
原文摘要 · Abstract (English)
In this paper, we propose to improve end-to-end (E2E) spoken language understand (SLU) in an RNN transducer model (RNN-T) by incorporating a joint self-conditioned CTC automatic speech recognition (ASR) objective. Our proposed model is akin to an E2E differentiable cascaded model which performs ASR and SLU sequentially and we ensure that the SLU task is conditioned on the ASR task by having CTC self conditioning. This novel joint modeling of ASR and SLU improves SLU performance significantly over just using SLU optimization. We further improve the performance by aligning the acoustic embeddings of this model with the semantically richer BERT model. Our proposed knowledge transfer strategy makes use of a bag-of-entity prediction layer on the aligned embeddings and the output of this is used to condition the RNN-T based SLU decoding. These techniques show significant improvement over several strong baselines and can perform at par with large models like Whisper with significantly fewer parameters.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。