用渐进交错的语音文本训练提升大模型语音翻译能力
Scheduled Interleaved Speech-Text Training for Speech-to-Speech Translation with LLMs
- 训练时交替使用语音和文本单位，逐步减少文本比例
- 在CVSS数据集上微调LLaMA3.2-1B，提升有限语料下的翻译性能
- 特别适合低资源语言，缓解大模型从文本到语音的模态适配难题
语音到语音翻译（S2ST）借助大语言模型（LLMs）取得进展，这些模型在离散语音单元上进行微调。然而，从文本到语音的模态适应仍是挑战，因LLMs仅在纯文本数据上训练，且语音-语音数据有限。为解决此问题，本文提出调度式交错语音-文本训练：训练中使用交错的语音-文本单元（单词级对齐），并随训练进程逐步降低文本比例，实现从文本到语音的渐进模态适应。我们在CVSS数据集上微调了LLaMA3.2-1B模型进行S2ST实验，结果表明该方法显著提升翻译性能，尤其在训练数据较少的语言上表现更优。
原文摘要 · Abstract (English)
Speech-to-speech translation (S2ST) has been advanced with large language models (LLMs), which are fine-tuned on discrete speech units. In such approaches, modality adaptation from text to speech has been an issue. LLMs are trained on text-only data, which presents challenges to adapt them to speech modality with limited speech-to-speech data. To address the training difficulty, we propose scheduled interleaved speech--text training in this study. We use interleaved speech--text units instead of speech units during training, where aligned text tokens are interleaved at the word level. We gradually decrease the ratio of text as training progresses, to facilitate progressive modality adaptation from text to speech. We conduct experimental evaluations by fine-tuning LLaMA3.2-1B for S2ST on the CVSS dataset. We show that the proposed method consistently improves the translation performances, especially for languages with limited training data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。