提出新型音频注意力机制，提升自监督学习性能
ASDA: Audio Spectrogram Differential Attention Mechanism for Self-Supervised Representation Learning
- 采用双Softmax与差分系数设计，减少无效注意力分配
- 在多个数据集上达到最新最好结果，如AS-2M达49.0% mAP
- 适合音频分类、关键词识别等自监督学习任务
近期音频自监督表征学习中，标准Transformer架构虽为主流，但其注意力机制常将部分权重分配给无关信息，可能削弱模型判别能力。为此，我们提出一种差分注意力机制，通过引入双Softmax操作和合理调校的差分系数，有效缓解无效注意力分配问题。实验表明，所提ASDA模型在多个基准上取得当前最优性能：音频分类任务中，AS-2M数据集达49.0% mAP，AS20K数据集达41.5% mAP；关键词检测任务中，SPC-2数据集准确率达98.3%；环境声音分类任务中，ESC-50数据集准确率达96.1%。这些结果验证了ASDA在音频任务中的有效性，为更广泛应用奠定基础。
原文摘要 · Abstract (English)
In recent advancements in audio self-supervised representation learning, the standard Transformer architecture has emerged as the predominant approach, yet its attention mechanism often allocates a portion of attention weights to irrelevant information, potentially impairing the model's discriminative ability. To address this, we introduce a differential attention mechanism, which effectively mitigates ineffective attention allocation through the integration of dual-softmax operations and appropriately tuned differential coefficients. Experimental results demonstrate that our ASDA model achieves state-of-the-art (SOTA) performance across multiple benchmarks, including audio classification (49.0% mAP on AS-2M, 41.5% mAP on AS20K), keyword spotting (98.3% accuracy on SPC-2), and environmental sound classification (96.1% accuracy on ESC-50). These results highlight ASDA's effectiveness in audio tasks, paving the way for broader applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。