提出高效神经转录模型,联合提升语音识别与翻译性能。
HENT-SRT: Hierarchical Efficient Neural Transducer with Self-Distillation for Joint Speech Recognition and Translation
- 分层结构分离识别与翻译任务,更好处理词序问题。
- 在阿拉伯语、西班牙语、中文数据集上超越现有转录模型。
- 通过自蒸馏和空白惩罚,提升翻译质量且训练更高效。
神经转录器(NT)在语音流式识别中表现优异,但在联合语音识别与翻译(ST)时仍面临词序重排和性能下降的挑战,难以媲美基于注意力机制的编码器-解码器(AED)模型。现有基于NT的ST方法还存在训练计算成本高的问题。为此,本文提出HENT-SRT(面向语音识别与翻译的分层高效神经转录器),通过分解识别与翻译任务以更好应对词序问题;采用带CTC一致性正则化的自蒸馏策略,在保持识别性能的同时增强翻译鲁棒性;同时引入下采样分层编码器、无状态预测器和剪枝的转录损失,降低训练复杂度;并在解码阶段加入空白惩罚,减少误删,提升翻译质量。在阿拉伯语、西班牙语和中文三个对话数据集上验证,该方法在NT模型中达到新最佳性能,并显著缩小与AED系统间的差距。
原文摘要 · Abstract (English)
Neural transducers (NT) provide an effective framework for speech streaming, demonstrating strong performance in automatic speech recognition (ASR). However, the application of NT to speech translation (ST) remains challenging, as existing approaches struggle with word reordering and performance degradation when jointly modeling ASR and ST, resulting in a gap with attention-based encoder-decoder (AED) models. Existing NT-based ST approaches also suffer from high computational training costs. To address these issues, we propose HENT-SRT (Hierarchical Efficient Neural Transducer for Speech Recognition and Translation), a novel framework that factorizes ASR and translation tasks to better handle reordering. To ensure robust ST while preserving ASR performance, we use self-distillation with CTC consistency regularization. Moreover, we improve computational efficiency by incorporating best practices from ASR transducers, including a down-sampled hierarchical encoder, a stateless predictor, and a pruned transducer loss to reduce training complexity. Finally, we introduce a blank penalty during decoding, reducing deletions and improving translation quality. Our approach is evaluated on three conversational datasets Arabic, Spanish, and Mandarin achieving new state-of-the-art performance among NT models and substantially narrowing the gap with AED-based systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。