用单模型处理多层语音编码，提升语音生成一致性。
Llama-Mimi: Exploring the Limits of Flattened Speech Language Modeling
- 将多层语音离散码本序列扁平化，用单个Transformer建模。
- 在声学一致性任务上表现最佳，超越分层结构模型。
- 适合关注语音生成质量与模型简洁性的研究者。
语音语言模型（SpeechLM）通过分词语音捕捉语义与声学信息。当基于残差向量量化（RVQ）的神经音频编解码器作为音频分词器时，每时间步产生多个离散令牌，形成固有的多层次表示。以往工作通常采用分层架构来捕捉这种结构，但近期自然语言处理领域趋向简化架构，减少归纳偏置，采用更简洁可扩展的单个Transformer。本文提出Llama-Mimi，将Mimi神经音频编解码器生成的多层RVQ令牌扁平化为单一序列，并使用Transformer解码器进行自回归建模。实验表明，Llama-Mimi在多数任务上优于基于因果掩码序列（CSM）的分层模型，在声学一致性方面达到最优性能。模型、代码与语音样本均已公开。
原文摘要 · Abstract (English)
Speech Language Models (SpeechLMs) model tokenized speech to capture both semantic and acoustic information. When neural audio codecs based on Residual Vector Quantization (RVQ) are used as audio tokenizers, they produce multiple discrete tokens per time step, yielding inherently multi-level representations. To process these multi-level tokens together, prior work typically adopts hierarchical architectures to capture this structure. In contrast, recent progress in NLP has progressively reduced architectural inductive biases, moving toward simpler and more scalable single-Transformer architectures. In this work, we propose Llama-Mimi, which flattens multi-level RVQ tokens produced by the Mimi neural audio codec into a single sequence and models them autoregressively with a Transformer decoder. We show that Llama-Mimi outperforms a CSM-based hierarchical model on most tasks and achieves the best performance on acoustic consistency. Our models, code, and speech samples are publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。