用动态时间规整对齐语音与文本表示，提升端到端语音翻译精度。
DTW-Align: Bridging the Modality Gap in End-to-End Speech Translation with Dynamic Time Warping Alignment
- 引入动态时间规整（DTW）在训练中对齐语音与文本嵌入。
- 在6个语言方向中5个低资源场景表现更优，速度更快。
- 无需依赖额外对齐工具，适用于多语言且不依赖词级对齐。
端到端语音翻译（E2E-ST）旨在直接将源语音翻译为目标文本，跳过中间转录步骤。语音与文本模态间的表征差异导致了‘模态鸿沟’问题。现有方法通过词或符号级别对齐来缓解该问题，但需依赖特定语言的对齐工具。虽有研究采用最近邻相似性搜索对齐嵌入，但准确性不足。本文提出在训练中使用动态时间规整（DTW）对齐语音与文本嵌入。实验表明，该方法有效缩小模态差距，在多个语言方向上实现更精准对齐，达到可比的翻译性能，同时显著提升效率。尤其在低资源设置下，5/6语言方向优于先前方法。
原文摘要 · Abstract (English)
End-to-End Speech Translation (E2E-ST) is the task of translating source speech directly into target text bypassing the intermediate transcription step. The representation discrepancy between the speech and text modalities has motivated research on what is known as bridging the modality gap. State-of-the-art methods addressed this by aligning speech and text representations on the word or token level. Unfortunately, this requires an alignment tool that is not available for all languages. Although this issue has been addressed by aligning speech and text embeddings using nearest-neighbor similarity search, it does not lead to accurate alignments. In this work, we adapt Dynamic Time Warping (DTW) for aligning speech and text embeddings during training. Our experiments demonstrate the effectiveness of our method in bridging the modality gap in E2E-ST. Compared to previous work, our method produces more accurate alignments and achieves comparable E2E-ST results while being significantly faster. Furthermore, our method outperforms previous work in low resource settings on 5 out of 6 language directions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。