用大模型同时优化语音识别和翻译，提升语音翻译效果。
Speech Translation Refinement using Large Language Models
- 用大模型联合优化语音识别和翻译结果
- 在多个数据集上显著提升翻译准确率
- 适合需要高质量语音翻译的场景
大型语言模型（LLMs）在各类语言任务中展现出强大能力。受文本到文本翻译优化成功的启发，本文研究如何通过引入联合优化流程，利用大模型提升语音翻译（ST）性能。通过在无需训练的上下文学习和参数高效微调两种场景下，联合优化语音翻译与自动语音识别（ASR）转录结果，显著提升了ST模型的表现。此外，本文还探索了在上下文感知微调场景中，文档级上下文对优化效果的影响。在MuST-C和CoVoST 2数据集上的实验，涵盖七项翻译任务，验证了该方法的有效性，使用了GPT-3.5-turbo、LLaMA3-8B和Mistral-12B等主流大模型。进一步分析表明，联合优化转录与翻译优于仅优化翻译；引入文档级上下文可显著提升优化效果。代码与数据集已开源。
原文摘要 · Abstract (English)
Recent advancements in large language models (LLMs) have demonstrated their remarkable capabilities across various language tasks. Inspired by the success of text-to-text translation refinement, this paper investigates how LLMs can improve the performance of speech translation by introducing a joint refinement process. Through the joint refinement of speech translation (ST) and automatic speech recognition (ASR) transcription via LLMs, the performance of the ST model is significantly improved in both training-free in-context learning and parameter-efficient fine-tuning scenarios. Additionally, we explore the effect of document-level context on refinement under the context-aware fine-tuning scenario. Experimental results on the MuST-C and CoVoST 2 datasets, which include seven translation tasks, demonstrate the effectiveness of the proposed approach using several popular LLMs including GPT-3.5-turbo, LLaMA3-8B, and Mistral-12B. Further analysis further suggests that jointly refining both transcription and translation yields better performance compared to refining translation alone. Meanwhile, incorporating document-level context significantly enhances refinement performance. We release our code and datasets on GitHub.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。