用1.09亿对音视频数据训练，支持长短不一音频的跨模态模型
SLAP: Scalable Language-Audio Pretraining with Variable-Duration Audio and Multi-Objective Training
- 融合对比、自监督与字幕生成三类损失，单阶段训练提升细粒度特征学习
- 在10900万对数据上训练，支持任意时长音频输入
- 在零样本音频分类和音视频检索任务中刷新纪录，适合多场景应用
对比语言-音频预训练（CLAP）在学习语义丰富的音频表征方面取得了显著成果，并被广泛应用于各类音频任务。然而，现有CLAP模型存在三大局限：一是通常在较小数据集上训练，仅含数百万音频样本；二是仅支持短且固定时长的音频，难以适应真实场景中时长可变的音频；三是标准对比损失基于全局表示，不利于细粒度特征的学习。为此，我们提出可扩展的语言-音频预训练（SLAP），将语言-音频预训练规模扩展至1.09亿个音视频对，支持可变时长音频输入，并引入多目标训练机制。SLAP在单阶段训练中统一对比损失、自监督损失与字幕生成损失，促进更丰富的密集音频表征学习。该模型在音视频检索和零样本音频分类任务上达到新的最先进性能，在多个基准测试中表现优异。
原文摘要 · Abstract (English)
Contrastive language-audio pretraining (CLAP) has achieved notable success in learning semantically rich audio representations and is widely adopted for various audio-related tasks. However, current CLAP models face several key limitations. First, they are typically trained on relatively small datasets, often comprising a few million audio samples. Second, existing CLAP models are restricted to short and fixed duration, which constrains their usage in real-world scenarios with variable-duration audio. Third, the standard contrastive training objective operates on global representations, which may hinder the learning of dense, fine-grained audio features. To address these challenges, we introduce Scalable Language-Audio Pretraining (SLAP), which scales language-audio pretraining to 109 million audio-text pairs with variable audio durations and incorporates multiple training objectives. SLAP unifies contrastive loss with additional self-supervised and captioning losses in a single-stage training, facilitating the learning of richer dense audio representations. The proposed SLAP model achieves new state-of-the-art performance on audio-text retrieval and zero-shot audio classification tasks, demonstrating its effectiveness across diverse benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。