动态调整语音分帧，用更少令牌实现更高保真度语音重建
Beyond Fixed Frames: Dynamic Character-Aligned Speech Tokenization
- 通过字符级软对齐实现可变帧率分词
- 在低帧率下仍保持高质量语音还原，令牌数减少40%以上
- 支持解码时直接控制令牌时长，适合语音生成与压缩场景
神经音频编解码器是现代对话式语音技术的核心，将连续语音转换为离散令牌序列以供大语言模型处理。然而，现有编解码器通常采用固定帧率，时间上均匀分配令牌，导致序列过长。本文提出DyCAST，一种基于软字符级对齐和显式时长建模的动态字符对齐语音分词器，可在训练中学习将令牌与字符级语言单元关联，并支持无需对齐的解码，解码时可直接控制令牌时长。为提升低帧率下的语音重构质量，进一步引入检索增强解码机制，在不增加码率的前提下提高重建保真度。实验表明，DyCAST在保持竞争性语音重建质量和下游任务性能的同时，显著减少了令牌数量。代码与模型检查点将公开于 https://github.com/lucadellalib/dycast。
原文摘要 · Abstract (English)
Neural audio codecs are at the core of modern conversational speech technologies, converting continuous speech into sequences of discrete tokens that can be processed by LLMs. However, existing codecs typically operate at fixed frame rates, allocating tokens uniformly in time and producing unnecessarily long sequences. In this work, we introduce DyCAST, a Dynamic Character-Aligned Speech Tokenizer that enables variable-frame-rate tokenization through soft character-level alignment and explicit duration modeling. DyCAST learns to associate tokens with character-level linguistic units during training and supports alignment-free inference with direct control over token durations at decoding time. To improve speech resynthesis quality at low frame rates, we further introduce a retrieval-augmented decoding mechanism that enhances reconstruction fidelity without increasing bitrate. Experiments show that DyCAST achieves competitive speech resynthesis quality and downstream performance while using significantly fewer tokens than fixed-frame-rate codecs. Code and checkpoints will be released publicly at https://github.com/lucadellalib/dycast.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。