用合成音频模式预训练音频编码器，避免真实数据隐私问题。
Pre-training with Synthetic Patterns for Audio
- 用掩码自编码器重构合成声谱图，学习通用特征表示。
- 13个音频任务测试中表现媲美AudioSet-2M预训练模型。
- 适合关注数据安全与高效预训练的研究者使用。
本文提出一种使用合成模式而非真实音频数据进行音频编码器预训练的框架。核心包括两个部分：一是掩码自编码器（MAE），通过随机遮蔽输入并重建来学习数据内在规律；二是合成数据，其无隐私和版权风险。由于MAE不依赖内容语义，合成声谱图同样可作为有效输入。我们在17个合成数据集上开展13项音频任务的实验，验证了该框架的有效性。结果表明，该方法在性能上可比肩基于AudioSet-2M预训练的模型，并部分超越基于图像的预训练方法。
原文摘要 · Abstract (English)
In this paper, we propose to pre-train audio encoders using synthetic patterns instead of real audio data. Our proposed framework consists of two key elements. The first one is Masked Autoencoder (MAE), a self-supervised learning framework that learns from reconstructing data from randomly masked counterparts. MAEs tend to focus on low-level information such as visual patterns and regularities within data. Therefore, it is unimportant what is portrayed in the input, whether it be images, audio mel-spectrograms, or even synthetic patterns. This leads to the second key element, which is synthetic data. Synthetic data, unlike real audio, is free from privacy and licensing infringement issues. By combining MAEs and synthetic patterns, our framework enables the model to learn generalized feature representations without real data, while addressing the issues related to real audio. To evaluate the efficacy of our framework, we conduct extensive experiments across a total of 13 audio tasks and 17 synthetic datasets. The experiments provide insights into which types of synthetic patterns are effective for audio. Our results demonstrate that our framework achieves performance comparable to models pre-trained on AudioSet-2M and partially outperforms image-based pre-training methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。