让语音大模型跨语言训练更高效，少数据也能学好多语种对话。
Efficient Training for Cross-lingual Speech Language Models
- 用离散语音标记+持续预训练，实现语音与语言的跨模态对齐。
- 在多语言对话任务上表现优异，仅用少量语音数据即达高精度。
- 适合想做多语种语音交互、资源有限的研究者或开发者。
当前大语言模型主要聚焦文本模态。为实现更自然的人机交互，语音大模型正兴起，但构建有效的端到端跨语言语音大模型仍面临数据稀缺和多语言扩展难的问题。本文提出跨语言语音语言模型（CSLM），一种基于离散语音标记的高效跨语言语音大模型训练方法。通过持续预训练实现跨模态与跨语言对齐，并采用语音-文本交错的链式模态生成方式进行指令微调，提升细粒度模态对齐，从而改善生成质量并降低延迟。CSLM无需海量语音数据即可实现多语言可扩展性，在跨模态任务、单语言对话任务及跨语言对话任务上的评估均验证了其强大的跨模态对齐能力与通用任务表现。（代码已公开：https://github.com/ictnlp/CSLM）
原文摘要 · Abstract (English)
Currently, large language models (LLMs) predominantly focus on the text modality. To enable more natural human-AI interaction, speech LLMs are emerging, but building effective end-to-end speech LLMs remains challenging due to limited data and the difficulty in expanding to more languages. In this paper, we introduce Cross-lingual Speech Language Model (CSLM), an efficient training method for cross-lingual speech LLMs based on discrete speech tokens. We propose a novel alignment strategy that achieves cross-modal and cross-lingual alignment through continual pre-training. By conducting instruction fine-tuning following a speech-text interleaved chain-of-modality generation process, we enhance modal alignment at a finer granularity, thereby improving generation quality and reducing latency. CSLM aligns different modalities and languages simultaneously without the need for massive speech data, thus exhibiting good language scalability. Evaluations on cross-modal tasks, mono-lingual conversational tasks, and cross-lingual conversational tasks demonstrate CSLM's strong cross-modal alignment capabilities and general task abilities. (Code is available at: https://github.com/ictnlp/CSLM)
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。