用掩码频谱图预测构建音频表征，少数据高效训练
Audio-JEPA: Joint-Embedding Predictive Architecture for Audio Representation Learning
- 用ViT模型预测掩码频谱块的高层特征，不重建原始音频
- 仅用1/5数据量达到wav2vec 2.0和data2vec水平性能
- 无需调参，适合音频表征学习初学者快速复现
基于联合嵌入预测架构（JEPA）范式，我们提出专为音频设计的Audio-JEPA。该方法使用简单的Vision Transformer骨干网络，预测掩码梅尔频谱图块的潜在表示，而非重建原始音频。在未标注的AudioSet片段（10秒，32kHz）上进行预训练，采用随机频谱块掩码策略。在X-ARES基准上评估，涵盖语音、音乐和环境声任务。尽管实现是原模型向音频的直接迁移，结果仍可媲美wav2vec 2.0和data2vec，且训练数据不足其五分之一，且无需超参数调优。所有代码与预训练权重将公开于GitHub。
原文摘要 · Abstract (English)
Building on the Joint-Embedding Predictive Architecture (JEPA) paradigm, a recent self-supervised learning framework that predicts latent representations of masked regions in high-level feature spaces, we propose Audio-JEPA (Audio Joint-Embedding Predictive Architecture), tailored specifically for audio data. Audio-JEPA uses a simple Vision Transformer backbone to predict latent representations of masked spectrogram patches rather than reconstructing raw audio. We pre-train on unlabeled AudioSet clips (10s, 32kHz) with random patch masking on mel-spectrograms. We evaluate on the X-ARES suite covering speech, music, and environmental sound tasks. Although our implementation is a straightforward translation of the original model to audio, the results still show comparable performance to wav2vec 2.0 and data2vec while using less than one-fifth of their training data and with no hyper-parameter tuning. All code and pretrained checkpoints will be released on GitHub.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。