用检索增强提升语音翻译术语准确率，实时生成更精准。
RASST: Retrieval-Augmented Simultaneous Speech Translation
- 构建轻量级跨模态检索器，分块提取术语提示。
- 术语准确率提升近40%，整体翻译质量最高增3 BLEU。
- 适合需要高精度术语翻译的实时场景，如会议同传。
同时语音翻译从部分语音输入中增量生成目标文本。近年来的语音大模型显著提升了同步语音翻译（SST）质量，但仍难以处理罕见和领域特定术语。检索增强在自动语音识别和神经机器翻译中已有效，但扩展至SST面临挑战：检索需在部分语音输入下快速且准确，且模型必须判断何时、是否使用检索到的术语进行增量生成。本文提出检索增强的同步语音翻译（RASST），解决上述问题。针对部分输入下的准确跨模态检索，RASST训练了一个轻量级语音-文本检索器，通过多尺度检索为语音大模型生成分块术语提示。为正确使用这些提示，我们合成训练数据，教会语音大模型判断每个检索项是否及何时使用。在ACL 60/60开发集和ESO测试集上的实验表明，RASST将术语准确率提高近40%，整体翻译质量最高提升3 BLEU点，计算开销几乎不变。
原文摘要 · Abstract (English)
Simultaneous speech translation produces target text incrementally from partial speech input. Recent speech large language models have markedly improved SST quality but still struggle with rare and domain-specific terminology. Retrieval augmentation has helped in automatic speech recognition and neural machine translation, but extending it to SST is non-trivial: retrieval must be fast and accurate under partial speech, and the model must decide whether and when to apply retrieved terms during incremental generation. We propose Retrieval-Augmented Simultaneous Speech Translation (RASST), which addresses both challenges. For accurate cross-modal retrieval under partial input, RASST trains a lightweight speech-text retriever that produces chunkwise terminology hints for the Speech LLM via multi-scale retrieval. To use these hints correctly, we synthesize training data that teaches the Speech LLM to decide whether and when to apply each retrieved term. Experiments on ACL 60/60 dev set and the ESO test set show that RASST improves terminology accuracy by nearly 40% and overall translation quality by up to 3 BLEU points, with negligible computational overhead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。