用快慢分支结构让语音增强模型更省算力，超低延迟下仍保持效果。
Modulating State Space Model with SlowFast Framework for Compute-Efficient Ultra Low-Latency Speech Enhancement
- 快分支用状态空间模型做实时增强，慢分支低频分析环境并动态调节快分支。
- 在2毫秒延迟下算力降低70%，62.5微秒延迟时仅需100M MACs/s。
- 适合对延迟和算力敏感的实时语音增强场景，如智能音箱、耳机。
基于深度学习的语音增强（SE）方法在满足低延迟要求时面临显著计算挑战，因需处理更多帧数。本文提出慢-快框架（SlowFast），专门应对低延迟增强中的高算力问题。该框架包含一个以低帧率分析声学环境的慢分支，以及一个以所需高帧率在时域执行语音增强的快分支。具体地，快分支采用状态空间模型，其状态转移过程由慢分支动态调制。在需2毫秒算法延迟的语音增强任务上，使用Voice Bank + Demand数据集的实验表明，相比参数相当的单分支基线网络，本方法算力降低70%且性能无损。进一步地，借助该框架，我们实现算法延迟仅62.5微秒（16 kHz采样率下一采样点），算力为100 M MACs/s，PESQ-NB得分为3.12，SISNR为16.62。
原文摘要 · Abstract (English)
Deep learning-based speech enhancement (SE) methods often face significant computational challenges when needing to meet low-latency requirements because of the increased number of frames to be processed. This paper introduces the SlowFast framework which aims to reduce computation costs specifically when low-latency enhancement is needed. The framework consists of a slow branch that analyzes the acoustic environment at a low frame rate, and a fast branch that performs SE in the time domain at the needed higher frame rate to match the required latency. Specifically, the fast branch employs a state space model where its state transition process is dynamically modulated by the slow branch. Experiments on a SE task with a 2 ms algorithmic latency requirement using the Voice Bank + Demand dataset show that our approach reduces computation cost by 70% compared to a baseline single-branch network with equivalent parameters, without compromising enhancement performance. Furthermore, by leveraging the SlowFast framework, we implemented a network that achieves an algorithmic latency of just 62.5 μs (one sample point at 16 kHz sample rate) with a computation cost of 100 M MACs/s, while scoring a PESQ-NB of 3.12 and SISNR of 16.62.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。