arXiv:2506.16738cs.CLcs.AI2025-06

通过语义语音重建蒸馏,让语音标记更贴合语言模型,提升跨模态统一建模效果。

LM-SPT: LM-Aligned Semantic Distillation for Speech Tokenization

  • 用语义语音重建实现间接蒸馏,避免固定时间对齐,增强语义一致性。
  • 在语音识别与文生语音任务中均优于现有方法,且保持编码器级语音重建质量。
  • 适合追求高质量语音-文本统一建模的研究者与开发者使用。

随着语音语言模型(SLMs)的快速发展,离散语音标记已成为语音与文本间的核心接口,支持跨模态统一建模。现有语音标记方法旨在将语义信息从低层声学特征中分离,以更好地对齐语言模型(LMs)。以往方法采用自监督学习(SSL)教师模型(如HuBERT)提取语义表示,并将其蒸馏至语义量化器,以抑制声学冗余并捕捉内容相关潜在结构。然而,这些标记器通常帧率较高,生成的标记序列远长于对应文本,阻碍与预训练语言模型的无缝集成。尽管近期方法尝试通过均匀平均池化降低标记率,但可能过度平滑含内容区域,稀释结构信息,从而限制语言模型对齐。为此,本文提出LM-SPT,一种基于语义语音重建蒸馏的语音标记方法。不同于直接通过池化匹配师生特征,LM-SPT仅用语义标记重建语音,并最小化原始波形与重建波形经冻结的、对齐语言模型的语音编码器提取表征之间的差异。这种间接监督避免了刚性的时间对齐,鼓励在较低帧率下生成更具语义一致性的独立语义单元。实验表明,当应用于语音语言模型进行自动语音识别和文本到语音任务时,所提方法始终优于先前的语义增强型语音标记器,且不牺牲编码器层级的语音重建保真度。

原文摘要 · Abstract (English)

With the rapid progress of speech language models (SLMs), discrete speech tokens have emerged as a core interface between speech and text, enabling unified modeling across modalities. Recent speech tokenization approaches aim to isolate semantic information from low-level acoustics to better align with language models (LMs). In particular, previous methods use self-supervised learning (SSL) teachers such as HuBERT to extract semantic representations, which are then distilled into a semantic quantizer to suppress acoustic redundancy as well as capture content-related latent structures. However, these tokenizers often operate at relatively high frame rates, producing token sequences significantly longer than their textual counterparts and hindering seamless integration with pretrained LMs. Although recent methods attempt to reduce the token rate by applying uniform average pooling to SSL features, this can over-smooth content-bearing regions and dilute the structural information, thereby potentially limiting the LM alignment. To address this, we propose LM-SPT, an LM-aligned speech tokenization method based on semantic speech-resynthesis distillation. Instead of directly matching teacher and student features via pooling, LM-SPT resynthesizes speech from semantic tokens only and minimizes the discrepancy between representations extracted from the original and resynthesized waveforms using a frozen, LM-aligned speech encoder. This indirect supervision avoids rigid temporal alignment and encourages dedicated semantic units that are more semantically aligned with LMs under reduced frame rates. Experimental results show that the proposed LM-SPT consistently outperforms previous semantic-enhanced speech tokenizers when applied to SLMs for the tasks of automatic speech recognition and text-to-speech, even without compromising the speech reconstruction fidelity at the codec level.

语音标记语言模型蒸馏跨模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。