将语音输入融入翻译专用大模型，实现10种语言的语音转译。
From TOWER to SPIRE: Adding the Speech Modality to a Translation-Specialist LLM
- 用语音离散化技术将语音当作文本语言训练，融合到多语言模型中。
- 仅用42.5小时语音数据即实现跨语言语音转译与文本翻译。
- 适合需要多模态翻译能力的研究者与开发者使用。
我们提出Spire，一种具备语音输入处理能力的多语言语言模型，可将英语语音转译为10种其他语言，并支持双向文本翻译。通过语音离散化和仅42.5千小时语音数据的持续预训练，将语音模态整合至现有多语言模型中。该方法借鉴多语言模型预训练框架，将离散化语音视为额外翻译语言，既赋予模型语音能力，又保持原有文本翻译性能。实验表明，以更少数据即可实现语音能力集成，验证了将离散语音作为附加语言在语言模型适配中的可行性。代码与模型已开源。
原文摘要 · Abstract (English)
We introduce Spire, a speech-augmented language model (LM) capable of both translating and transcribing speech input from English into 10 other languages as well as translating text input in both language directions. Spire integrates the speech modality into an existing multilingual LM via speech discretization and continued pre-training using only 42.5K hours of speech. In particular, we adopt the pretraining framework of multilingual LMs and treat discretized speech input as an additional translation language. This approach not only equips the model with speech capabilities, but also preserves its strong text-based performance. We achieve this using significantly less data than existing speech LMs, demonstrating that discretized speech input integration as an additional language is feasible during LM adaptation. We make our code and models available to the community.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。