arXiv:2509.00503cs.CLeess.AS2025-09被引 1

用熵值动态压缩语音表示,提升语义建模效率。

Entropy-based Coarse and Compressed Semantic Speech Representation Learning

  • 基于预测熵自动聚合语音片段,实现可调压缩粒度。
  • 在语音识别等任务中表现优于或相当密集标记序列。
  • 适合需要高效语义表示的语音下游任务开发者。

离散语音表征学习在声学与语义建模中日益受到关注。现有方法通常将16 kHz波形编码为每秒25或50个离散标记,但语音每秒仅传达2至5个词,这种细粒度标记引入冗余,影响下游训练与推理效率。此外,高频率语义表征主要捕捉音素级信息,而语义理解未必需要如此精细的粒度。为此,本文提出一种基于熵的动态聚合框架,用于学习压缩的语义语音表征。首先在大规模无标签数据上通过下一项预测预训练语言模型,以捕捉频繁的标记模式;随后利用预测熵自适应确定聚合边界,并通过交叉注意力模块融合每段内信息。通过调整熵阈值,可灵活控制表征的粒度与压缩率。在语音识别、语音到文本翻译及语音转换任务上的实验表明,压缩后的表征性能达到或超过密集标记序列,验证了该方法的有效性。

原文摘要 · Abstract (English)

Discrete speech representation learning has recently attracted increasing interest in both acoustic and semantic modeling. Existing approaches typically encode 16 kHz waveforms into discrete tokens at a rate of 25 or 50 tokens per second. However, given that speech generally conveys only 2 to 5 words per second, such fine-grained tokenization introduces redundancy and hinders efficiency in downstream training and inference. Moreover, semantic speech representations at this frequency primarily capture phonetic-level information, while semantic understanding may not require such detailed token-level resolution. To address these limitations, we propose an entropy-based dynamic aggregation framework for learning compressed semantic speech representations. A speech language model is first pre-trained via next-token prediction on large-scale unlabeled data to capture frequent token patterns. Predictive entropy is then used to adaptively determine aggregation boundaries, followed by a cross-attention module that fuses information within each segment. By adjusting the entropy threshold, the granularity and compression ratio of the representations can be flexibly controlled. Experiments on ASR, speech-to-text translation, and voice conversion tasks demonstrate that the compressed representations perform on par with or better than dense token sequences, demonstrating the effectiveness of the proposed approach.

语音表征语义压缩熵控制动态聚合

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。