用语音压缩匹配文本，提升流式语音翻译准确率与速度
CTC-GMM: CTC guided modality matching for fast and accurate streaming speech translation
- 用CTC将语音转为紧凑嵌入，与机器翻译文本对齐
- 在FLEURS上提升准确率13.9%，CoVoST2上提升6.4%
- 适合追求低延迟高精度的实时语音翻译场景
流式语音翻译模型若基于大量源语言语音与目标语言文本配对数据训练，可实现高精度与低延迟。然而，目标语言文本标签常因人工标注成本过高而为伪标签。本文提出连接时序分类引导的模态匹配（CTC-GMM）方法，通过利用大规模机器翻译文本数据增强流式语音翻译模型。该方法使用CTC将语音序列压缩为紧凑嵌入序列，并与对应文本序列对齐，从而可复用机器翻译语料中的{源-目标}语言文本对进一步优化模型。在FLEURS和CoVoST2上的实验表明，该方法相对提升翻译准确率分别为13.9%和6.4%，同时在GPU上加速解码59.7%。
原文摘要 · Abstract (English)
Models for streaming speech translation (ST) can achieve high accuracy and low latency if they're developed with vast amounts of paired audio in the source language and written text in the target language. Yet, these text labels for the target language are often pseudo labels due to the prohibitive cost of manual ST data labeling. In this paper, we introduce a methodology named Connectionist Temporal Classification guided modality matching (CTC-GMM) that enhances the streaming ST model by leveraging extensive machine translation (MT) text data. This technique employs CTC to compress the speech sequence into a compact embedding sequence that matches the corresponding text sequence, allowing us to utilize matched {source-target} language text pairs from the MT corpora to refine the streaming ST model further. Our evaluations with FLEURS and CoVoST2 show that the CTC-GMM approach can increase translation accuracy relatively by 13.9% and 6.4% respectively, while also boosting decoding speed by 59.7% on GPU.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。