用声谱图的视觉与频域双重特性,实现轻量级语音增强。
Dual-View Predictive Diffusion: Lightweight Speech Enhancement via Spectrogram-Image Synergy
- 将声谱图视为视觉纹理和频域信号,双视角协同处理。
- 参数仅需SOTA模型35%,推理计算量减少40%。
- 无需微调即可提升生成质量,适合资源受限场景。
扩散模型近期在语音增强(SE)任务中达到新基准。然而,现有基于得分的模型通常将声谱图当作普通2D图像处理,忽略音频固有的结构稀疏性,导致谱表示效率低下且计算复杂度高。为此,我们提出极轻量级的双视角预测扩散模型DVPD,创新性地在训练与推理阶段均利用声谱图作为视觉纹理与物理频域表示的双重属性。训练时,通过频率自适应非均匀压缩(FANC)编码器优化频谱利用率,保留关键低频谐波并剔除高频冗余;同时引入轻量级图像感知模块(LISA),以极少开销捕获视觉特征。推理时,提出无需训练的无损增益(TLB)策略,利用相同双视角先验提升生成质量。大量实验表明,DVPD在多个基准上达到顶尖性能,参数量仅为当前轻量级模型PGUSE的35%,推理乘加操作(MACs)降低至40%。结果凸显其在高保真语音质量与极致架构效率间的优异平衡。代码与音频样例见匿名网站:{https://anonymous.4open.science/r/dvpd_demo-E630}。
原文摘要 · Abstract (English)
Diffusion models have recently set new benchmarks in Speech Enhancement (SE). However, most existing score-based models treat speech spectrograms merely as generic 2D images, applying uniform processing that ignores the intrinsic structural sparsity of audio, which results in inefficient spectral representation and prohibitive computational complexity. To bridge this gap, we propose DVPD, an extremely lightweight Dual-View Predictive Diffusion model, which uniquely exploits the dual nature of spectrograms as both visual textures and physical frequency-domain representations across both training and inference stages. Specifically, during training, we optimize spectral utilization via the Frequency-Adaptive Non-uniform Compression (FANC) encoder, which preserves critical low-frequency harmonics while pruning high-frequency redundancies. Simultaneously, we introduce a Lightweight Image-based Spectro-Awareness (LISA) module to capture features from a visual perspective with minimal overhead. During inference, we propose a Training-free Lossless Boost (TLB) strategy that leverages the same dual-view priors to refine generation quality without any additional fine-tuning. Extensive experiments across various benchmarks demonstrate that DVPD achieves state-of-the-art performance while requiring only 35% of the parameters and 40% of the inference MACs compared to SOTA lightweight model, PGUSE. These results highlight DVPD's superior ability to balance high-fidelity speech quality with extreme architectural efficiency. Code and audio samples are available at the anonymous website: {https://anonymous.4open.science/r/dvpd_demo-E630}
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。