用可学习全局令牌增强Whisper，实现更紧凑的音文嵌入
WavLink: Compact Audio-Text Embeddings with a Global Whisper Token
- 在Whisper编码器中加入可学习全局令牌，联合训练文本编码器
- 8倍压缩嵌入尺寸，性能下降极小，检索效果达顶尖水平
- 适合需要高效音文匹配的应用，如语音检索与零样本分类
Whisper已成为大型音文模型中提取通用音频特征的事实标准，通常30秒音频片段被表示为1500帧特征并投影至大语言模型。相比之下，音文嵌入模型如基于CLAP的模型主要依赖其他音频编码器（如HTS-AT、PaSST），未有效利用Whisper。我们提出WavLink，一种紧凑的音文嵌入模型，通过在Whisper编码器中引入可学习的全局令牌，并与文本编码器联合训练。通过系统研究设计选择，包括预训练文本编码器、损失函数、训练模式和数据混合策略，我们确定了取得最佳检索性能的配置。采用两阶段训练流程，结合马特里什卡式监督，三种模型尺寸下实现8倍嵌入压缩，性能损失微小。WavLink在AIR-Bench上的多项选择题和零样本分类任务中也表现优异。
原文摘要 · Abstract (English)
Whisper has become the de-facto encoder for extracting general-purpose audio features in large audio-language models, where a 30-second clip is typically represented by 1500 frame features projected into an LLM. In contrast, audio-text embedding models like CLAP-based models have largely relied on alternative audio encoders (e.g., HTS-AT, PaSST), and have not leveraged Whisper effectively. We present WavLink, a compact audio-text embedding model that augments Whisper encoder with a learnable global token, trained jointly with a text encoder. Through a systematic study of design choices, including pretrained text encoders, loss functions, training modes, and data mixtures, we identify configurations that yield state-of-the-art retrieval performance. Our two-stage training recipe across three model sizes, combined with Matryoshka-style supervision, improves scalability, enabling 8x smaller embeddings with minimal performance drop. WavLink also demonstrates competitive performance on AIR-Bench with MCQs and zero-shot classification.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。