一种可同时生成连续与离散音频表示的统一编码器，兼顾高压缩比与高保真。
CoDiCodec: Unifying Continuous and Discrete Compressed Representations of Audio
- 用有限标量量化和新dropout技术，统一生成连续嵌入（~11 Hz）与离散标记（2.38 kbps）。
- 在相似码率下，重建音频质量超越现有连续与离散自编码器。
- 支持自回归与并行解码，适合不同生成任务，灵活性强。
高效地将音频信号压缩到低维潜在空间对潜在生成建模至关重要。然而，现有自编码器通常迫使用户在连续嵌入与离散标记之间二选一。此外，在保持音频保真度的前提下实现高压缩比仍具挑战。我们提出CoDiCodec，一种新型音频自编码器，通过总结嵌入高效编码全局特征，并从同一训练模型中同时生成约11 Hz的压缩连续嵌入与2.38 kbps的离散标记，为下游生成任务提供前所未有的灵活性。该方法基于有限标量量化（FSQ）及一种新颖的FSQ-dropout技术，且无需额外损失项，仅使用单一一致性损失即可端到端训练。CoDiCodec支持自回归解码和一种新的并行解码策略，后者在音频质量与解码速度上均表现更优。在相近比特率下，其重建音频质量优于现有连续与离散自编码器。本工作实现了音频压缩的统一范式，弥合了连续与离散生成建模之间的鸿沟。
原文摘要 · Abstract (English)
Efficiently representing audio signals in a compressed latent space is critical for latent generative modelling. However, existing autoencoders often force a choice between continuous embeddings and discrete tokens. Furthermore, achieving high compression ratios while maintaining audio fidelity remains a challenge. We introduce CoDiCodec, a novel audio autoencoder that overcomes these limitations by both efficiently encoding global features via summary embeddings, and by producing both compressed continuous embeddings at ~ 11 Hz and discrete tokens at a rate of 2.38 kbps from the same trained model, offering unprecedented flexibility for different downstream generative tasks. This is achieved through Finite Scalar Quantization (FSQ) and a novel FSQ-dropout technique, and does not require additional loss terms beyond the single consistency loss used for end-to-end training. CoDiCodec supports both autoregressive decoding and a novel parallel decoding strategy, with the latter achieving superior audio quality and faster decoding. CoDiCodec outperforms existing continuous and discrete autoencoders at similar bitrates in terms of reconstruction audio quality. Our work enables a unified approach to audio compression, bridging the gap between continuous and discrete generative modelling paradigms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。