arXiv:2605.14231cs.LGcs.AI2026-05被引 2

AudioMosaic通过结构化掩码构建对比学习音频表示,提升跨域泛化能力。

AudioMosaic: Contrastive Masked Audio Representation Learning

论文配图:AudioMosaic: Contrastive Masked Audio Representation Learning
图 1 · 摘自论文原文
  • 用时频块结构化掩码生成正样本对,降低内存占用并支持大批次训练。
  • 在多个基准上优于生成式方法,线性探测和微调均达当前最佳性能。
  • 适合需要强泛化性的音频理解任务,尤其适用于多场景语音与声学环境。

音频自监督学习(SSL)旨在从大规模无标签音频数据中学习通用表示。尽管近期进展主要由生成式重建目标驱动,对比学习方法仍较少被探索,部分原因在于难以设计有效的音频增强策略以及对比预训练所需的较大批量。本文提出基于对比学习的音频编码器 AudioMosaic。在预训练阶段,AudioMosaic 通过对频谱图块施加结构化的时频掩码构建正样本对,降低了内存消耗并实现高效的大型批次训练。相较于生成式方法,AudioMosaic 编码器学习到更具判别性的语句级表示,在不同数据集、领域和声学条件下表现出更强的迁移能力。大量实验表明,AudioMosaic 在多个标准音频基准上在线性探测和微调设置下均达到当前最优表现。进一步实验显示,将预训练的 AudioMosaic 编码器集成至音频-语言模型中,可提升音频-语言任务性能。代码已公开于 GitHub 仓库。

原文摘要 · Abstract (English)

Audio self-supervised learning (SSL) aims to learn general-purpose representations from large-scale unlabeled audio data. While recent advances have been driven mainly by generative reconstruction objectives, contrastive approaches remain less explored, partly due to the difficulty of designing effective audio augmentations and the large batch sizes required for contrastive pre-training. We introduce \textbf{AudioMosaic}, a contrastive learning-based audio encoder for general audio understanding. During pre-training, AudioMosaic constructs positive pairs by applying structured time-frequency masking to spectrogram patches, which reduces memory usage and enables efficient large-batch training. Compared with generative approaches, the AudioMosaic encoder learns more discriminative utterance-level representations that demonstrate strong transferability across datasets, domains, and acoustic conditions. Extensive experiments show that AudioMosaic achieves state-of-the-art performance on several standard audio benchmarks under both linear probing and fine-tuning. We further show that integrating the pretrained AudioMosaic encoder into audio-language models improves performance on audio-language tasks. The code is publicly available in our \href{https://github.com/HanxunH/AudioMosaic}{GitHub repository}.

音频表示对比学习自监督

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。