轻量级音视频语音分离模型,兼顾高精度与低延迟。
Efficient Audio-Visual Speech Separation with Discrete Lip Semantics and Multi-Scale Global-Local Attention
- 用离散唇语语义令牌替代原始视频,降低视觉特征复杂度
- 多尺度全局-局部注意力机制提升跨时序依赖捕捉效率
- 参数减少50%以上,推理速度提升6倍,适合实时部署
音视频语音分离(AVSS)利用视觉线索在嘈杂环境中提取目标语音,表现优异。但现有方法参数量大、计算开销高,难以应用于仅作为后续处理预处理的场景。为此,我们提出高效AVSS模型Dolphin。视觉特征提取采用双路径轻量级视频编码器DP-LipCoder,将唇动转化为对齐音频的离散语义标记;音频分离部分设计轻量级编码器-解码器结构,每层引入全局-局部注意力(GLA)块以高效捕获多尺度依赖。在三个基准数据集上的实验表明,Dolphin不仅超越当前最先进(SOTA)模型的分离质量,还在效率上实现显著提升:参数量减少超过50%,乘加操作(MACs)降低2.4倍以上,GPU推理速度提升6倍以上。结果表明,Dolphin为实际场景中的高性能音视频语音分离提供了可落地的解决方案。代码与演示页已公开于http://cslikai.cn/Dolphin/。
原文摘要 · Abstract (English)
Audio-visual speech separation (AVSS) methods leverage visual cues to extract target speech and have demonstrated strong separation quality in noisy acoustic environments. However, these methods usually involve a large number of parameters and require high computational cost, which is unacceptable in many applications where speech separation serves as only a preprocessing step for further speech processing. To address this issue, we propose an efficient AVSS method, named Dolphin. For visual feature extraction, we develop DP-LipCoder, a dual-path lightweight video encoder that transforms lip-motion into discrete audio-aligned semantic tokens. For audio separation, we construct a lightweight encoder-decoder separator, in which each layer incorporates a global-local attention (GLA) block to efficiently capture multi-scale dependencies. Experiments on three benchmark datasets showed that Dolphin not only surpassed the current state-of-the-art (SOTA) model in separation quality but also achieved remarkable improvements in efficiency: over 50% fewer parameters, more than 2.4x reduction in MACs, and over 6x faster GPU inference speed. These results indicate that Dolphin offers a practical and deployable solution for high-performance AVSS in real-world scenarios. Our code and demo page are publicly available at http://cslikai.cn/Dolphin/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。