CleanCodec只保留人耳感知重要的语音信息，高效压缩音频。
CleanCodec: Efficient and Robust Speech Tokenization via Perceptually Guided Encoding
- 将语音编码视为选择性信息瓶颈，只保留关键声学特征。
- 每秒仅12.5个词元，显著提升语音可懂度与说话人相似性。
- 适合对效率和质量要求高的语音生成与转换任务。
神经音频编解码器是语音处理流程的关键组件，将音频压缩为离散词元以支持下游建模。然而，现有编解码器难以在重建质量与词元效率之间取得平衡，常编码背景噪声等感知无关信息，牺牲了语言和声学有意义的内容。本文将语音分词重构为选择性信息瓶颈问题，提出CleanCodec——一种去噪音频编解码器，学习仅编码感知重要特征并丢弃不可察觉信息。该方法仅需每秒12.5个词元，达到当前最优的词元效率，在说话人相似性和语音可懂度上显著优于现有方法。下游文本转语音与语音转换任务评估显示性能提升，推理速度最快可达17倍加速，体现显著效率优势。
原文摘要 · Abstract (English)
Neural audio codecs are a key component of speech processing pipelines, compressing audio into discrete tokens for downstream modeling. However, existing codecs struggle to balance reconstruction quality with token efficiency, often encoding perceptually irrelevant information such as background noise and recording artifacts at the expense of linguistically and acoustically meaningful content. We reframe audio tokenization as a selective information bottleneck problem and propose CleanCodec, a denoising audio codec which learns to encode only perceptually important features and discard imperceptible information. At just 12.5 tokens per second, CleanCodec achieves state-of-the-art tokenization efficiency, substantially outperforming existing codecs in speaker similarity and speech intelligibility. Evaluations on downstream text-to-speech and voice conversion tasks further demonstrate improved performance and up to 17x faster inference, highlighting significant efficiency gains.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。