0.35kbps下实现高保真音乐压缩，突破低码率音质瓶颈
MuCodec: Ultra Low-Bitrate Music Codec
- 分离声学与语义特征，用流匹配生成梅尔频谱
- 0.35kbps下主观评分超现有方法，客观指标最优
- 适合低带宽音乐传输与生成，开源可运行
音乐编码器是音频编码研究的重要方向，超低码率压缩对音乐传输与生成具有重要意义。由于音乐背景复杂、人声丰富，仅建模语义或声学信息难以有效还原包含人声和背景的音乐。为此，我们提出MuCodec，专门针对超低码率下的音乐压缩与重建任务。MuCodec采用MuEncoder提取声学与语义特征，通过RVQ离散化，并利用流匹配获得梅尔-变分自编码器（Mel-VAE）特征。音乐通过预训练的Mel-VAE解码器与HiFi-GAN重建。该方法在0.35kbps超低码率及1.35kbps高码率下均能实现高保真音乐重建，在主观与客观评估中均达到当前最佳表现。代码与演示：https://xuyaoxun.github.io/MuCodec_demo/
原文摘要 · Abstract (English)
Music codecs are a vital aspect of audio codec research, and ultra low-bitrate compression holds significant importance for music transmission and generation. Due to the complexity of music backgrounds and the richness of vocals, solely relying on modeling semantic or acoustic information cannot effectively reconstruct music with both vocals and backgrounds. To address this issue, we propose MuCodec, specifically targeting music compression and reconstruction tasks at ultra low bitrates. MuCodec employs MuEncoder to extract both acoustic and semantic features, discretizes them with RVQ, and obtains Mel-VAE features via flow-matching. The music is then reconstructed using a pre-trained MEL-VAE decoder and HiFi-GAN. MuCodec can reconstruct high-fidelity music at ultra low (0.35kbps) or high bitrates (1.35kbps), achieving the best results to date in both subjective and objective metrics. Code and Demo: https://xuyaoxun.github.io/MuCodec_demo/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。