用程序生成音频预训练，不用真实录音也能学好声音表示。
From Physics to Representation: Audio Learning with Synthetic Pre-training via Procedural Generation
- 用声学基元和组合规则实时生成波形，训练Transformer掩码自编码器。
- 在ESC-50等4个数据集上表现优异，最高准确率达97.03%。
- 单卡20分钟内完成预训练，且表征可线性解码物理特征。
自监督学习推动了多媒体分析中的音频表征发展。然而，现有以数据为中心的方法依赖大规模真实音频数据集，增加了训练成本、数据整理负担和隐私风险。为此，我们提出AudioPG，一种在预训练阶段完全不使用真实音频录制的程序化合成框架。AudioPG基于Transformer的掩码自编码器，在实时生成的波形上进行训练，这些波形由基本声学基元和组合规则生成。该编码器在真实音频基准测试中表现出色：在ESC-50上达到90.60%准确率，FSD50K上mAP为0.546，UrbanSound8K上达88.17%，Speech Commands V2上达97.03%。值得注意的是，预训练仅需单张GPU运行20分钟以内。潜在空间分析显示，基频、相对强度等物理因素在正交子空间中显现，使表征具备线性可解性。结果表明，当缺乏大规模数据集时，程序化合成是一种高效且可解释的预训练信号。代码已开源：https://github.com/Freyliu0516/audioPG。
原文摘要 · Abstract (English)
Self-supervised learning advances audio representation for multimedia analysis. However, prevailing data-centric approaches rely on massive real-world corpora, increasing training costs, curation burdens, and privacy barriers. To address this, we present AudioPG, a procedural synthesis framework eliminating real audio recordings during pre-training. AudioPG trains a Transformer-based masked autoencoder on waveforms generated on-the-fly from basic acoustic primitives and composition rules. The encoder transfers effectively to real audio benchmarks, achieving 90.60% accuracy on ESC-50, 0.546 mAP on FSD50K, 88.17% on UrbanSound8K, and 97.03% on Speech Commands V2. Notably, pre-training completes in under 20 minutes on a single GPU. Latent space analysis reveals physical factors, including fundamental frequency and relative intensity, emerge in orthogonal subspaces, making representations linearly decodable. These results establish procedural synthesis as an efficient, interpretable pre-training signal when large-scale corpora are unavailable. Our code is available at: https://github.com/Freyliu0516/audioPG.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。