轻量级语音分离模型，实时区分车内不同位置乘客说话
DualSep: A Light-weight dual-encoder convolutional recurrent network for real-time in-car speech separation
- 结合数字信号处理与双分支神经网络，分步提取空间和频谱特征
- 仅0.83M参数、0.39实时因子，在普通CPU上实现低延迟分离
- 适合车载语音系统部署，支持实时流式与非流式模式
深度学习与语音交互技术的发展推动了人车交互的进步。分布式麦克风阵列广泛应用于车内场景，可精准捕捉不同语音区域的乘客声音。然而，通道数量增加，叠加车载系统计算资源有限和低延迟要求，给多通道语音分离带来挑战。为此，我们提出一种轻量级框架，将数字信号处理（DSP）与神经网络（NN）级联。利用固定波束成形（BF）降低计算开销，独立向量分析（IVA）提供空间先验。采用双编码器进行双分支建模，空间编码器捕捉空间线索，谱编码器保留频谱信息，实现空间-频谱融合。所提系统支持流式与非流式模式。实验表明，该系统在多种指标上表现优越。仅需0.83M参数和0.39实时因子（RTF），在英特尔酷睿i7（2.6GHz）CPU上即可有效分离不同语音区域的声音。演示视频见https://honee-w.github.io/DualSep/。
原文摘要 · Abstract (English)
Advancements in deep learning and voice-activated technologies have driven the development of human-vehicle interaction. Distributed microphone arrays are widely used in in-car scenarios because they can accurately capture the voices of passengers from different speech zones. However, the increase in the number of audio channels, coupled with the limited computational resources and low latency requirements of in-car systems, presents challenges for in-car multi-channel speech separation. To migrate the problems, we propose a lightweight framework that cascades digital signal processing (DSP) and neural networks (NN). We utilize fixed beamforming (BF) to reduce computational costs and independent vector analysis (IVA) to provide spatial prior. We employ dual encoders for dual-branch modeling, with spatial encoder capturing spatial cues and spectral encoder preserving spectral information, facilitating spatial-spectral fusion. Our proposed system supports both streaming and non-streaming modes. Experimental results demonstrate the superiority of the proposed system across various metrics. With only 0.83M parameters and 0.39 real-time factor (RTF) on an Intel Core i7 (2.6GHz) CPU, it effectively separates speech into distinct speech zones. Our demos are available at https://honee-w.github.io/DualSep/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。