用音节分词让语音模型更高效，训练快一倍、算力省五倍。
Scaling Spoken Language Models with Syllabic Speech Tokenization
- 用音节级别分词替代高帧率语音分词，降低序列长度。
- 在多个语音理解任务上表现不输甚至超越原有方法，训练时间减半，计算量降为1/5。
- 适合追求高效长上下文语音建模的研究者和工业应用开发者。
语音语言模型（SLMs）通常将语音离散化为从自监督学习（SSL）模型中提取的高帧率标记。由于最成功的语言模型基于Transformer架构，处理这些长标记序列时自注意力机制开销巨大，其复杂度随序列长度呈平方增长。近期一项SSL研究提出在音节层级进行语音分词，更具可解释性且显著压缩标记长度（4-5 Hz）。但其在语音语言建模中的潜力尚未充分探索。本文首次系统研究音节分词在语音语言建模中的应用，在多种语音理解（SLU）基准上评估模型性能，并改变训练数据规模。结果表明，音节标记可在保持或超越高帧率标记性能的同时，大幅降低训练与推理成本：训练时间减少超过2倍，浮点运算量（FLOPs）减少5倍。研究强调音节级语言建模是实现高效长上下文语音模型的有前景路径。
原文摘要 · Abstract (English)
Spoken language models (SLMs) typically discretize speech into high-frame-rate tokens extracted from SSL speech models. As the most successful LMs are based on the Transformer architecture, processing these long token streams with self-attention is expensive, as attention scales quadratically with sequence length. A recent SSL work introduces acoustic tokenization of speech at the syllable level, which is more interpretable and potentially more scalable with significant compression in token lengths (4-5 Hz). Yet, their value for spoken language modeling is not yet fully explored. We present the first systematic study of syllabic tokenization for spoken language modeling, evaluating models on a suite of SLU benchmarks while varying training data scale. Syllabic tokens can match or surpass the previous high-frame rate tokens while significantly cutting training and inference costs, achieving more than a 2x reduction in training time and a 5x reduction in FLOPs. Our findings highlight syllable-level language modeling as a promising path to efficient long-context spoken language models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。