让语音编码器按需调整帧率，节省冗余计算。
CodecSlime: Temporal Redundancy Compression of Neural Speech Codec via Dynamic Frame Rate
- 动态帧率机制，按语音内容密度分配编码资源。
- 40赫兹下字错误率降32%，同等码率更优。
- 无需修改模型，适配各类神经语音编码器。
神经语音编码器广泛应用于音频压缩及下游任务。现有主流编码器采用固定帧率（FFR），对等长片段均分配相同数量的标记，但语音在时间上的信息密度本就不均匀，导致稳态段如长元音和静音中大量标记被浪费。为解决此问题，本文提出CodecSlime，首次实现神经语音编码器上支持动态帧率（DFR）的插件式冗余压缩方法。该方法无监督且与架构无关，结合两项关键创新：ScheDFR（用于推理适应）与Melt-and-Cool（用于训练适应）。在典型VQ-GAN编码器基础上，以40赫兹动态帧率（约600 bps）运行时，重构字错误率（WER）相比传统FFR基线降低最高达32%，其他指标亦具竞争力。此外，单模型可灵活支持多帧率推理，在对应帧率下性能始终优于固定帧率模型。音频样例见https://acadarmeria.github.io/codecslime/。
原文摘要 · Abstract (English)
Neural speech codecs have been widely used in audio compression and various downstream tasks. Current mainstream codecs are fixed-frame-rate (FFR), which allocate the same number of tokens to every equal-duration slice. However, speech is inherently non-uniform in temporal information density. As a result, many tokens are wasted on steady-state segments like long vowels and silences. To address this mismatch, we present CodecSlime, a plugin-style method for compressing temporal redundancy through supporting dynamic frame rate (DFR) on neural speech codecs for the first time. Our method is unsupervised and architecture-agnostic, combining two key innovations, ScheDFR and Melt-and-Cool, for adapting inference and training, respectively. When integrated into a typical VQ-GAN codec backbone and operating at 40 Hz DFR ($\approx$ 600 bps), the reconstruction WER of CodecSlime is reduced by up to 32% relative to conventional FFR baselines with the same model architecture and similar bitrates, while other metrics are also competitive. CodecSlime also enables flexible trade-offs between reconstruction quality and bitrate: a single model supports inference at multiple frame rates and consistently outperforms FFR models at the corresponding frame rates. Audio samples are available at https://acadarmeria.github.io/codecslime/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。