通过混合多语言语音单元，让语音模型跨语言理解与对话。
Cross-Lingual Interleaving for Speech Language Models
- 用无文本监督的跨语言语音标记交织训练模型。
- 在360M和1B参数模型上提升单语语义准确率和跨语言连贯性。
- 适合做多语言语音模型研究，资源开源可复现。
语音语言模型（SLMs）旨在直接从语音中学习语言能力，使书写资源有限的语言也能接入自然语言处理技术。然而，由于缺乏口语评估基准和训练数据，进展长期集中于英语，跨语言学习困难。本文提出一种无需文本监督的跨语言语音标记交织方法，并发布包含约42,000小时的英法双语训练数据集TinyStories，以及基于GPT-4合成的英法双语语音StoryCloze和TopicCloze评估基准。在360M和1B参数的SLMs上，相同训练令牌预算下，交织方法提升了单语语义准确率，实现了稳健的跨语言续写能力，并增强了跨语言隐状态对齐。结果表明，该方法是构建多语言语音模型、实现跨语言理解与对话的简单且可扩展路径。所有资源将开源以支持复现。
原文摘要 · Abstract (English)
Spoken Language Models (SLMs) aim to learn linguistic competence directly from speech using discrete units, widening access to Natural Language Processing (NLP) technologies for languages with limited written resources. However, progress has been largely English-centric due to scarce spoken evaluation benchmarks and training data, making cross-lingual learning difficult. We present a cross-lingual interleaving method that mixes speech tokens across languages without textual supervision. We also release an EN-FR training dataset, TinyStories (~42k hours), together with EN-FR spoken StoryCloze and TopicCloze benchmarks for cross-lingual semantic evaluation, both synthetically generated using GPT-4. On 360M and 1B SLMs under matched training-token budgets, interleaving improves monolingual semantic accuracy, enables robust cross-lingual continuation, and strengthens cross-lingual hidden-state alignment. Taken together, these results indicate that cross-lingual interleaving is a simple, scalable route to building multilingual SLMs that understand and converse across languages. All resources will be made open-source to support reproducibility.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。