用语音转写文本引导翻译,让语音大模型更准地跨语言翻译。
Chain-of-Thought Prompting for Speech Translation
- 先转写语音为文本,再用文本和语音共同引导翻译,像思维链一样逐步推理。
- 在6个中英互译任务上,比单纯用语音提示提升2.4个BLEU点。
- 适合想提升语音翻译准确率的研究者或开发者使用。
大型语言模型(LLMs)在语言理解与生成方面取得了显著进展。基于文本型LLM的成功,近期研究将这些模型适配为使用语音嵌入进行提示,形成了在自动语音识别(ASR)和自动语音翻译(AST)任务中表现优异的Speech-LLM模型。本文提出一种新方法:利用ASR转写文本作为提示,驱动基于编码器-解码器结构的Megatron-T5构建的Speech-LLM进行语音翻译。该模型通过两步流程实现:先解码语音生成ASR转写,再结合转写文本与编码后的语音进行提示,模拟思维链(Chain-of-Thought, CoT)推理。采用低秩适应(LoRA)对T5 LLM进行微调,在性能上优于全模型微调。实验结果表明,所提CoT提示方法显著提升AST性能,在6个En→X或X→En翻译任务上平均提升2.4 BLEU点,相比另一种将ASR与翻译转写拼接预测的CoT方法,平均高出2 BLEU点。
原文摘要 · Abstract (English)
Large language models (LLMs) have demonstrated remarkable advancements in language understanding and generation. Building on the success of text-based LLMs, recent research has adapted these models to use speech embeddings for prompting, resulting in Speech-LLM models that exhibit strong performance in automatic speech recognition (ASR) and automatic speech translation (AST). In this work, we propose a novel approach to leverage ASR transcripts as prompts for AST in a Speech-LLM built on an encoder-decoder text LLM. The Speech-LLM model consists of a speech encoder and an encoder-decoder structure Megatron-T5. By first decoding speech to generate ASR transcripts and subsequently using these transcripts along with encoded speech for prompting, we guide the speech translation in a two-step process like chain-of-thought (CoT) prompting. Low-rank adaptation (LoRA) is used for the T5 LLM for model adaptation and shows superior performance to full model fine-tuning. Experimental results show that the proposed CoT prompting significantly improves AST performance, achieving an average increase of 2.4 BLEU points across 6 En->X or X->En AST tasks compared to speech prompting alone. Additionally, compared to a related CoT prediction method that predicts a concatenated sequence of ASR and AST transcripts, our method performs better by an average of 2 BLEU points.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。