用嵌入损失直接训练压缩音频语音分离模型，提速两倍且效果更好。
Speech Separation using Neural Audio Codecs with Embedding Loss
- 在压缩音频表示上直接使用嵌入损失，无需解码训练
- 训练速度提升2倍，计算成本降低，WSJ0-2mix上音质和清晰度更优
- 适合追求高效语音分离的开发者与系统集成者
神经音频编解码器通过高压缩表示革新了音频处理，使语音任务可在压缩域内执行，实现更快训练和更低推理开销。然而，现有方法仍依赖波形级损失函数，训练时需额外解码步骤。本文提出一种新型嵌入损失，直接作用于压缩音频表示，消除训练中的解码需求。我们在多个客观指标与感知评估方法（包括侵入式与非侵入式）上进行综合验证。结果表明，该方法在3个不同预训练编解码器下，于WSJ0-2mix数据集上实现2倍训练速度提升与计算成本降低，同时获得更优的DNSMOS与STOI表现。
原文摘要 · Abstract (English)
Neural audio codecs have revolutionized audio processing by enabling speech tasks to be performed on highly compressed representations. Recent work has shown that speech separation can be achieved within these compressed domains, offering faster training and reduced inference costs. However, current approaches still rely on waveform-based loss functions, necessitating unnecessary decoding steps during training. We propose a novel embedding loss for neural audio codec-based speech separation that operates directly on compressed audio representations, eliminating the need for decoding during training. To validate our approach, we conduct comprehensive evaluations using both objective metrics and perceptual assessment techniques, including intrusive and non-intrusive methods. Our results demonstrate that embedding loss can be used to train codec-based speech separation models with a 2x improvement in training speed and computational cost while achieving better DNSMOS and STOI performance on the WSJ0-2mix dataset across 3 different pre-trained codecs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。