将语音表示为音节结构,实现高效低码率的语音建模。
Sylber: Syllabic Embedding Representation of Speech from Raw Audio
- 自监督学习框架通过自身音节分割蒸馏生成音节嵌入。
- 每秒仅需4.27个音节标记,支持线性时间分割。
- 音节嵌入自然产生类别感知,适合语音建模与压缩。
音节是人类语言感知与生成中高效的组合单位,但现有神经语音表示缺乏此类结构,导致密集的标记序列难以高效处理。为此,我们提出Sylber模型,通过自监督学习框架,利用自身初始无监督音节分割进行蒸馏,生成具有清晰稳健音节结构的语音表示。该方法带来三大优势:1)可实现快速线性时间音节分割;2)音节标记效率高,平均每秒仅需4.27个标记;3)生成适合高效语音建模的新音位单位。所提分割方法鲁棒性强,无需调优即可泛化至域外数据和未见语言。通过训练端到端语音生成模型,仅用Sylber标记即可重建可懂语音,且比特率显著低于基线自监督标记。这表明模型能以极小信息损失将语音压缩为紧凑标记序列。最后,我们发现类别感知这一语言学现象在Sylber中自然涌现,使其嵌入空间更具类别性和稀疏性,进一步支持了其高效性。综上,我们提出一种新颖的自监督学习方法,将语音表示为音节,对高效语音标记与语音建模具有重要潜力。
原文摘要 · Abstract (English)
Syllables are compositional units of spoken language that efficiently structure human speech perception and production. However, current neural speech representations lack such structure, resulting in dense token sequences that are costly to process. To bridge this gap, we propose a new model, Sylber, that produces speech representations with clean and robust syllabic structure. Specifically, we propose a self-supervised learning (SSL) framework that bootstraps syllabic embeddings by distilling from its own initial unsupervised syllabic segmentation. This results in a highly structured representation of speech features, offering three key benefits: 1) a fast, linear-time syllable segmentation algorithm, 2) efficient syllabic tokenization with an average of 4.27 tokens per second, and 3) novel phonological units suited for efficient spoken language modeling. Our proposed segmentation method is highly robust and generalizes to out-of-domain data and unseen languages without any tuning. By training token-to-speech generative models, fully intelligible speech can be reconstructed from Sylber tokens with a significantly lower bitrate than baseline SSL tokens. This suggests that our model effectively compresses speech into a compact sequence of tokens with minimal information loss. Lastly, we demonstrate that categorical perception-a linguistic phenomenon in speech perception-emerges naturally in Sylber, making the embedding space more categorical and sparse than previous speech features and thus supporting the high efficiency of our tokenization. Together, we present a novel SSL approach for representing speech as syllables, with significant potential for efficient speech tokenization and spoken language modeling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。