动态调整语音分帧，用更少令牌实现更高自然度语音合成
Say More with Less: Variable-Frame-Rate Speech Tokenization via Adaptive Clustering and Implicit Duration Coding
- 根据语音局部相似性自适应分段，实现可变帧率编码
- 相比40赫兹固定帧率，减少23%令牌数且重建更自然
- 无需额外时长预测器，适合语音生成与合成任务
现有语音分词器通常每秒分配固定数量的令牌，忽略语音信号中信息密度和时间波动的差异。为解决这一问题，本文提出VARSTok——一种基于局部特征相似性的可变帧率语音分词器。其核心创新包括：(1) 一种考虑时间感知的密度峰值聚类算法，自适应地将语音分割为不等长单元；(2) 一种隐式时长编码机制，将内容与时间跨度整合进单一令牌索引，无需额外时长预测器。大量实验表明，VARSTok显著优于强基线模型。尤其在使用40赫兹固定帧率基准时，令牌数最多减少23%，同时实现更优的重建自然度。该方法在零样本文本到语音合成中也表现出更低的词错误率和更高的自然度。据我们所知，这是首个证明全动态可变帧率声学分词器可无缝集成至下游语音语言模型的工作。
原文摘要 · Abstract (English)
Existing speech tokenizers typically assign a fixed number of tokens per second, regardless of the varying information density or temporal fluctuations in the speech signal. This uniform token allocation mismatches the intrinsic structure of speech, where information is distributed unevenly over time. To address this, we propose VARSTok, a VAriable-frame-Rate Speech Tokenizer that adapts token allocation based on local feature similarity. VARSTok introduces two key innovations: (1) a temporal-aware density peak clustering algorithm that adaptively segments speech into variable-length units, and (2) a novel implicit duration coding scheme that embeds both content and temporal span into a single token index, eliminating the need for auxiliary duration predictors. Extensive experiments show that VARSTok significantly outperforms strong fixed-rate baselines. Notably, it achieves superior reconstruction naturalness while using up to 23% fewer tokens than a 40 Hz fixed-frame-rate baseline. VARSTok further yields lower word error rates and improved naturalness in zero-shot text-to-speech synthesis. To the best of our knowledge, this is the first work to demonstrate that a fully dynamic, variable-frame-rate acoustic speech tokenizer can be seamlessly integrated into downstream speech language models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。