将语音分离与压缩结合，用低比特离散表示实现多人语音分离传输。
CodeSep: Low-Bitrate Codec-Driven Speech Separation with Base-Token Disentanglement and Auxiliary-Token Serial Prediction
- 通过基令牌解耦与辅助令牌串行预测，实现语音分离与压缩联合建模。
- 仅传输基令牌，在1kbps下仍保持良好分离效果。
- 适合在线会议、对话存档等需高效语音传输的场景。
本文针对语音分离与语音压缩融合的新场景，旨在分离多说话人语音的同时生成适用于高效传输或存储的离散表示，适用于在线会议和对话归档。为此，我们提出CodeSep，一种基于编码器驱动的模型，可联合执行语音分离与低比特率压缩。CodeSep由基于残差向量量化（RVQ）的神经语音编码器、基令牌解耦（BTD）模块以及并行的辅助令牌串行预测（ATSP）模块组成。BTD模块将混合语音的梅尔频谱图解耦为每个说话人的基令牌，随后由ATSP模块串行预测辅助令牌，最终所有令牌通过编码器解码器重建分离波形。训练时，编码器的RVQ提供置换不变且基于教师强制的交叉熵损失监督。由于仅传输基令牌，CodeSep实现了低比特率压缩。实验结果表明，CodeSep在仅1 kbps的比特率下，性能优于基线方法。
原文摘要 · Abstract (English)
This paper targets a new scenario that integrates speech separation with speech compression, aiming to disentangle multiple speakers while producing discrete representations for efficient transmission or storage, with applications in online meetings and dialogue archiving. To address this scenario, we propose CodeSep, a codec-driven model that jointly performs speech separation and low-bitrate compression. CodeSep comprises a residual vector quantizer (RVQ)-based plain neural speech codec, a base-token disentanglement (BTD) module, and parallel auxiliary-token serial prediction (ATSP) modules. The BTD module disentangles mixed-speech mel-spectrograms into base tokens for each speaker, which are then refined by ATSP modules to serially predict auxiliary tokens, and finally, all tokens are decoded to reconstruct separated waveforms through the codec decoder. During training, the codec's RVQ provides supervision with permutation-invariant and teacher-forcing-based cross-entropy losses. As only base tokens are transmitted or stored, CodeSep achieves low-bitrate compression. Experimental results show that CodeSep attains satisfactory separation performance at only 1 kbps compared with baseline methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。