用局部注意力提升人声分离效率，速度提升44.5倍
Efficient Vocal Source Separation Through Windowed Sink Attention
- 用小窗口+注意力池化替代全时序自注意力
- 微调后音质损失仅8%，计算量降为原1/44.5
- 适合部署在移动端或实时场景的语音分离任务
当前先进的人声分离模型如Mel-Band-Roformer依赖全时序自注意力机制，导致计算开销随音频长度呈平方级增长。通过分析预训练模型发现，时间注意力模式高度局部化。基于此，我们提出窗口化注意力池（WSA），采用小时间窗口与注意力池化机制。实验证明，从原始检查点微调可恢复92%的原始信噪比改善（SDR）性能，同时将浮点运算量（FLOPs）降低44.5倍。代码与模型权重已开源，许可协议为MIT。
原文摘要 · Abstract (English)
State-of-the-art vocal separation models like Mel-Band-Roformer rely on full temporal self-attention mechanisms, where each temporal frame interacts with every other frames. This incurs heavy computational costs that scales quadratically with input audio length, motivating chunking and windowing approaches. Through analysis of a pre-trained vocal separation model, we discovered that temporal attention patterns are highly localized. Building on this insight, we replaced full attention with windowed sink attention (WSA) with small temporal attention window and attention sinks. We show empirically that fine-tuning from the original checkpoint recovers 92% of the original SDR performance while reducing FLOPs by 44.5x. We release our code and checkpoints under MIT license at https://github.com/smulelabs/windowed-roformer.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。