用检索示例提升语音翻译中生僻词的准确率
Optimizing Rare Word Accuracy in Direct Speech Translation with a Retrieval-and-Demonstration Approach
- 通过检索历史相似语料，让模型像上下文学习一样参考示例
- 使用真实示例提升生僻词翻译准确率17.6%，检索示例提升8.5%
- 支持跨模态检索，对陌生说话人更鲁棒，适合实际部署
直接语音翻译（Direct ST）模型在处理生僻词时表现不佳，错误翻译可能严重影响质量和用户信任。尽管神经模型因学习信号稀疏而难以处理此类词汇，但现实场景常可获取类似主题的历史录音翻译。为此，我们提出一种检索-示范方法来增强生僻词翻译准确性：首先将现有ST模型改造为可利用检索到的示例进行翻译，类似于上下文学习；其次开发跨模态（语音-语音、语音-文本、文本-文本）检索器定位合适示例。实验表明，标准ST模型可有效利用示例，使用真实示例时生僻词翻译准确率提升17.6%，使用检索示例时提升8.5%。此外，语音-语音检索优于其他模态，且对未见说话人更具鲁棒性。代码已公开于https://github.com/SiqiLii/Retrieve-and-Demonstration-ST。
原文摘要 · Abstract (English)
Direct speech translation (ST) models often struggle with rare words. Incorrect translation of these words can have severe consequences, impacting translation quality and user trust. While rare word translation is inherently challenging for neural models due to sparse learning signals, real-world scenarios often allow access to translations of past recordings on similar topics. To leverage these valuable resources, we propose a retrieval-and-demonstration approach to enhance rare word translation accuracy in direct ST models. First, we adapt existing ST models to incorporate retrieved examples for rare word translation, which allows the model to benefit from prepended examples, similar to in-context learning. We then develop a cross-modal (speech-to-speech, speech-to-text, text-to-text) retriever to locate suitable examples. We demonstrate that standard ST models can be effectively adapted to leverage examples for rare word translation, improving rare word translation accuracy over the baseline by 17.6% with gold examples and 8.5% with retrieved examples. Moreover, our speech-to-speech retrieval approach outperforms other modalities and exhibits higher robustness to unseen speakers. Our code is publicly available (https://github.com/SiqiLii/Retrieve-and-Demonstration-ST).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。