用三阶段方法逐步提升大模型的语音翻译能力。
PROST-LLM: Progressively Enhancing the Speech-to-Speech Translation Capability in LLMs
- 先用多任务学习和模态链训练初始模型。
- 自采样生成偏好对，实现无需人工标注的优化。
- 适合想在低资源下提升语音翻译的开发者。
尽管大语言模型在诸多任务中表现优异，但在语音到语音翻译（S2ST）领域的应用仍受限于数据稀缺。为此，我们提出PROST-LLM（Progressive Speech-to-Speech Translation），通过渐进式方法增强大模型的S2ST能力。首先，利用CVSS语料库对大模型进行微调，采用设计的三任务学习与模态链方法以提升初始性能；其次，基于微调后的模型，通过自采样与回译生成偏好对，避免依赖人工评估；最后，使用这些偏好对进行偏好优化，进一步提升模型的S2ST表现。大量实验验证了PROST-LLM在增强大模型语音翻译能力方面的有效性。
原文摘要 · Abstract (English)
Although Large Language Models (LLMs) excel in many tasks, their application to Speech-to-Speech Translation (S2ST) is underexplored and hindered by data scarcity. To bridge this gap, we propose PROST-LLM (PROgressive Speech-to-speech Translation) to enhance the S2ST capabilities in LLMs progressively. First, we fine-tune the LLMs with the CVSS corpus, employing designed tri-task learning and chain of modality methods to boost the initial performance. Then, leveraging the fine-tuned model, we generate preference pairs through self-sampling and back-translation without human evaluation. Finally, these preference pairs are used for preference optimization to enhance the model's S2ST capability further. Extensive experiments confirm the effectiveness of our proposed PROST-LLM in improving the S2ST capability of LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。