用改进的Transformer结构提升音频自监督表示能力
AudioMAE++: learning better masked audio representations with SwiGLU FFNs
- 采用带门控线性单元的新型变换器模块，优化音频建模
- 在AudioSet上预训练后，在10个下游任务中超越现有方法
- 参数量最多增加4倍仍保持良好扩展性，适合大规模音频任务
基于音频频谱图块的掩码自编码器（MAEs）已成为学习自监督音频表征的重要方法。尽管近期已有研究评估了音频MAE训练的关键方面，但多数方法仍使用传统的Transformer模块，而变换器社区已持续引入新架构进步。本文提出AudioMAE++，通过引入双增强机制——即带有门控线性单元的macaron风格变换器块，重构音频掩码自编码器。在AudioSet数据集上预训练后，所提AudioMAE++模型在10项多样化的下游任务中表现优于现有的基于MAE的方法，尤其在音频分类和语音基准测试中展现出优异性能。此外，该模型具有出色的可扩展性，在参数量最多达标准MAE基线4倍时仍保持更优效果。
原文摘要 · Abstract (English)
Masked Autoencoders (MAEs) trained on audio spectrogram patches have emerged as a prominent approach for learning self-supervised audio representations. While several recent papers have evaluated key aspects of training MAEs on audio data, the majority of these approaches still leverage vanilla transformer building blocks, whereas the transformer community has seen steady integration of newer architectural advancements. In this work, we propose AudioMAE++, a revamped audio masked autoencoder with two such enhancements, namely macaron-style transformer blocks with gated linear units. When pretrained on the AudioSet dataset, the proposed AudioMAE++ models outperform existing MAE based approaches on 10 diverse downstream tasks, demonstrating excellent performance on audio classification and speech-based benchmarks. The proposed AudioMAE++ models also demonstrate excellent scaling characteristics, outperforming directly comparable standard MAE baselines with up to 4x more parameters.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。