用多候选+自监督特征，让串行语音翻译更准
When End-to-End is Overkill: Rethinking Cascaded Speech-to-Text Translation
- 引入ASR多候选和自监督语音特征增强翻译
- 减少语音到文本映射中的语义偏差，提升精度
- 适合有大模型数据但追求高效准确的场景
尽管端到端语音到文本翻译已取得显著进展，但本文认为串行模型仍有价值，常被批评的错误传播问题主要源于语音域相似样本映射到文本域后出现的语义发散。通过引入ASR多候选结果与自监督语音特征，我们的方法使机器翻译模型能从多个语音样本中选择正确词汇，实现精准翻译。该策略有效抑制错误扩散，充分利用大规模ASR与MT数据集及预训练模型，同时缓解相关问题。
原文摘要 · Abstract (English)
Though end-to-end speech-to-text translation has been a great success, we argue that the cascaded speech-to-text translation model still has its place, which is usually criticized for the error propagation between automatic speech recognition (ASR) and machine translation (MT) models. In this paper, we explore the benefits of incorporating multiple candidates from ASR and self-supervised speech features into MT. Our analysis reveals that the primary cause of cascading errors stems from the increased divergence between similar samples in the speech domain when mapped to the text domain. By including multiple candidates and self-supervised speech features, our approach allows the machine translation model to choose the right words and ensure precise translation using various speech samples. This strategy minimizes error spread and takes advantage of large ASR and MT datasets, along with pre-trained ASR/MT models, while addressing associated issues.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。