DeCodec让音频编码器能分离语音与背景音，支持灵活控制。
DeCodec: Rethinking Audio Codecs as Universal Disentangled Representation Learners
- 通过正交子空间分解，将音频分出语音和背景音
- 实现语音增强、零样本变声、语音识别鲁棒性提升
- 适合语音处理、语音合成等需要成分控制的场景
通用音频编码器学习跨音频类型的纠缠表示，而部分专用编码器虽提供解耦表示，但仅限于语音。现实音频常包含混合语音与背景音，下游任务需选择性访问各成分。因此，我们重新思考音频编码器作为通用解耦表示学习者，以实现跨任务可控特征选择。为此，提出DeCodec，一种新型神经编码器，可将音频表示解耦为专用于语音和背景音的正交子空间，并在语音内进一步分解为语义与副语言成分。这种分层解耦支持灵活特征选择，使DeCodec成为多类音频应用的通用前端。技术上，基于编码器框架，引入两个关键创新：子空间正交投影模块，将输入分解为两个解耦正交子空间；表示交换训练过程，确保两子空间分别对应语音与背景音。这使得并行向量量化（RVQ）可独立量化语音与背景音成分。此外，采用语义引导语音RVQ，实现语义与副语言分解。实验表明，DeCodec在保持先进信号重建的同时，具备新能力：通过表示重组实现更优语音增强与噪声下零样本变声，利用纯净语义特征提升语音识别鲁棒性，在文本转语音中实现背景音可控保留或抑制。
原文摘要 · Abstract (English)
Universal audio codecs learn entangled representations across audio types, whereas some specific codecs offer decoupled representations but are limited to speech. Real-world audio, however, often contains mixed speech and background sounds, and downstream tasks require selective access to these components. Therefore, we rethink the audio codec as a universal disentangled representation learner to enable controllable feature selection across different audio tasks. To this end, we introduce DeCodec, a novel neural codec that learns to decouple audio representations into orthogonal subspaces dedicated to speech and background sound, and within speech, representations are further decomposed into semantic and paralinguistic components. This hierarchical disentanglement allows flexible feature selection, making DeCodec a universal front-end for multiple audio applications. Technically, built upon a codec framework, DeCodec incorporates two key innovations: a subspace orthogonal projection module that factorizes the input into two decoupled orthogonal subspaces, and a representation swap training procedure that ensures these two subspaces are correlate to the speech and background sound, respectively. These allows parallel RVQs to quantize speech and background sound components independently. Furthermore, we employ semantic guidance to the speech RVQ to achieve semantic and paralinguistic decomposition. Experimental results show that DeCodec maintains advanced signal reconstruction while enabling new capabilities: superior speech enhancement and effective one-shot voice conversion on noisy speech via representation recombination, improved ASR robustness through clean semantic features, and controllable background sound preservation/suppression in TTS. Demo Page: https://luo404.github.io/DeCodecV2/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。