通过多选学习增强音频自监督表示,提升复杂声音场景下的表征质量。
MATPAC++: Enhanced Masked Latent Prediction for Self-Supervised Audio Representation Learning
- 引入多选学习机制,显式建模音频预测中的歧义性
- 在AudioSet上微调达最新水平,下游任务综合表现最优
- 音乐数据专化训练时效率显著提升,适合音频表征任务
掩码潜在变量预测已成为自监督学习中音频与音乐表征学习的主流范式。尽管近期方法已展现强劲性能,但其输出端的预测模块作用仍被忽视,而该模块对解决预训练任务至关重要。尤其当音频包含多个声源时,需有效处理固有的不确定性。本文提出新改进:将多选学习(MCL)融入最近提出的MATPAC系统,增强其预测与无监督分类任务。我们在多个下游任务上通过线性探测与AudioSet微调进行评估,采用统一协议实现与顶尖方法的严格公平对比。结果表明,所提方法在微调后于AudioSet上达到领先水平,在整体下游任务中亦获最佳表现。此外,仅在音乐数据上训练时,模型性能达最新水平且效率大幅提升。
原文摘要 · Abstract (English)
Masked latent prediction has emerged as a leading paradigm in self-supervised learning (SSL), especially for general audio and music representation learning. While recent methods have demonstrated strong performance, the role of the predictor module used at the output of such SSL systems remains mainly overlooked, despite being crucial for solving the pretext task at hand. In particular, this module should be able to deal with the ambiguity inherent in audio content, especially when it is composed of multiple sound sources. This work proposes a novel enhancement: integrating Multiple Choice Learning (MCL) to explicitly model prediction ambiguity and improve representation quality. We build on top of the recently proposed MATPAC system, improving its prediction and unsupervised classification pretext tasks with MCL. We extensively evaluate our method, MATPAC++, through both linear probing across multiple downstream tasks and fine-tuning on AudioSet, employing a unified protocol that enables rigorous and fair comparisons with state-of-the-art SSL approaches. Results show that our proposal achieves state-of-the-art when fine-tuned on AudioSet and overall state-of-the-art scores on downstream tasks. Additionally, we examine domain specialisation by training exclusively on music data, where our model achieves state-of-the-art performance with significantly improved efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。