用渐进式知识蒸馏让语音增强模型又快又准
RT-SEMamba: Real-Time Speech Enhancement Mamba via Progressive Knowledge Distillation
- 基于因果时频状态空间块构建实时语音增强模型
- 1层学生模型在Voicebank-DEMAND上达到3.18 PESQ，提速2.75倍
- 适合需要低延迟高效率的实时语音处理场景
我们提出RT-SEMamba，一种基于因果时频Mamba块的全因果语音增强模型。与依赖不断增长的键值缓存的Transformer架构不同，Mamba每层仅传递固定大小的递归状态，实现内存和带宽高效的长序列推理。我们进一步提出一种渐进式知识蒸馏策略，将8层教师模型压缩为1层学生模型，联合蒸馏复杂频谱输出与中间表征。在Voicebank-DEMAND数据集上，8层RT-SEMamba在25毫秒算法延迟约束下达到3.32 PESQ；经蒸馏的1层学生模型相比原始1层基线从3.06提升至3.18 PESQ，且保持相同稳态实时因子（RTF），相较教师模型提速2.75倍。结果表明，结合渐进式知识蒸馏的状态空间模型可实现优秀的质量-延迟权衡，适用于实时语音增强任务。
原文摘要 · Abstract (English)
We present RT-SEMamba, a fully causal speech enhancement (SE) model built upon causal time-frequency Mamba blocks. Unlike Transformer-based architectures that rely on a growing key-value cache, Mamba propagates a fixed-size recurrent state per layer, enabling memory- and bandwidth-efficient long-form inference. We further introduce a progressive knowledge distillation (KD) strategy that compresses an 8-layer teacher into a shallow 1-layer student by jointly distilling complex spectral outputs and intermediate representations. On Voicebank-DEMAND, the 8-layer RT-SEMamba achieves 3.32 PESQ with a 25 ms algorithmic latency constraint, and the distilled 1-layer student improves over a naive 1-layer baseline from 3.06 to 3.18 PESQ while preserving the same steady-state RTF, delivering a 2.75x speedup over the teacher. These results demonstrate that state-space models with progressive KD provide a competitive quality-latency trade-off for real-time SE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。