开源7B级统一语音语言模型，支持语音识别与生成
OpusLM: A Family of Open Unified Speech Language Models
- 从纯文本模型出发，融合语音-文本对持续预训练
- 在213K小时语音数据上实现媲美甚至超越现有模型性能
- 全开源可复现，适合研究语音语言建模的开发者
本文提出开源统一语音语言模型（OpusLMs），包含最大规模达70亿参数的系列模型。模型以解码器单向文本语言模型为基础，利用213,000小时语音-文本配对数据及2920亿个纯文本标记进行持续预训练。实验表明，OpusLMs在语音识别、语音合成及纯文本能力方面达到与现有模型相当（或更优）的性能。技术上，论文阐述了分词设计、多流语言模型结构和多阶段训练策略。通过实验证明模型规模扩展的重要性及数据选择退火的影响。所有模型均基于公开材料构建，具备完全透明性，代码、数据、模型检查点与训练日志均已开放，以推动开放语音语言模型研究。
原文摘要 · Abstract (English)
This paper presents Open Unified Speech Language Models (OpusLMs), a family of open foundational speech language models (SpeechLMs) up to 7B. Initialized from decoder-only text language models, the OpusLMs are continuously pre-trained on 213K hours of speech-text pairs and 292B text-only tokens. We demonstrate our OpusLMs achieve comparable (or even superior) performance with existing SpeechLMs in speech recognition, speech synthesis, and text-only capabilities. Technically, this paper articulates our SpeechLM designs on tokenization, multi-stream language models, and multi-stage training strategies. We experimentally demonstrate the importance of model size scaling and the effect of annealing data selection. The OpusLMs are all built from publicly available materials and are fully transparent models. We release our code, data, checkpoints, and training logs to facilitate open SpeechLM research
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。