基于Phi4-MM的多模态模型，实现端到端语音翻译，效果优于现有基线。
SLM-S2ST: A multimodal language model for direct speech-to-speech translation
- 用音频变换器头预测延迟音频令牌，再通过流式声码器合成语音。
- 在CVSS-C数据集上性能显著超越同数据集训练的基线模型。
- 扩大数据和模型规模后，达到当前最佳水平，适合语音翻译研究者。
语音感知语言模型（SLM）已具备理解口语并生成文本响应的能力，但高效生成语音输出仍是挑战。本文提出SLM-S2ST，一种基于开源Phi4-MM模型的多模态语言模型，支持直接语音到语音翻译（S2ST）。该模型通过音频变换器头预测相对于文本令牌存在延迟的音频令牌，并利用流式声码器进行波形合成。在CVSS-C数据集上的实验表明，SLM-S2ST性能显著优于同数据集训练的现有基线模型。进一步扩大训练数据与模型规模后，其表现达到当前最先进（SOTA）水平。
原文摘要 · Abstract (English)
Speech-aware language models (LMs) have demonstrated capabilities in understanding spoken language while generating text-based responses. However, enabling them to produce speech output efficiently and effectively remains a challenge. In this paper, we present SLM-S2ST, a multimodal LM for direct speech-to-speech translation (S2ST), built on the open-source Phi4-MM model. SLM-S2ST extends its predecessor by generating translated speech using an audio transformer head that predicts audio tokens with a delay relative to text tokens, followed by a streaming vocoder for waveform synthesis. Our experimental results on the CVSS-C dataset demonstrate SLM-S2ST's superior performance, significantly surpassing existing baseline models trained on the same dataset. Furthermore, when we scale up the training data and the model size, SLM-S2ST reaches on-par performance with the current SOTA model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。