用原型聚类替代全局池化，提升音频多标签分类的探针性能
Unmute the Patch Tokens: Rethinking Probing in Multi-Label Audio Classification
- 提出二值原型探针，按类别聚合局部音片段信息
- 在13个数据集上优于线性与注意力探针，效果接近微调
- 适合评估自监督音频模型，无需昂贵微调
尽管探针冻结模型已成为标准评估范式，但音频自监督学习在AudioSet上仍以微调为主。根本原因在于全局池化造成信息瓶颈：[cls]标记丢弃了分散、局部音频事件的关键标记信息，这源于预训练目标（全局）与下游任务（局部）的不匹配。我们在13个数据集和6种基于频谱图的编码器上全面检验了这一瓶颈。提出二值原型探针：一种轻量级、简单的池化方法，通过学习原型实现类别级信息聚合。尽管结构简单，其性能显著优于线性与注意力探针。本工作确立了探针作为评估音频自监督模型的高效竞争范式，挑战了对高成本微调的依赖。
原文摘要 · Abstract (English)
Although probing frozen models has become a standard evaluation paradigm, self-supervised learning in audio defaults to fine-tuning when pursuing state-of-the-art on AudioSet. A key reason is that global pooling creates an information bottleneck causing linear probes to misrepresent the embedding quality: The $\texttt{cls}$-token discards crucial token information about dispersed, localized events in audio. This weakness is rooted in the mismatch between the pretraining objective (globally) and the downstream task (localized). Across a comprehensive benchmark of 13 datasets and 6 spectrogram-based encoders, we investigate the global pooling bottleneck. We introduce binarized prototypical probes: a lightweight and simple pooling method that learns prototypes to perform class-wise information aggregation. Despite its simplicity, our method notably outperforms linear and attentive probing. Our work establishes probing as a competitive and efficient paradigm for evaluating audio SSL models, challenging the reliance on costly fine-tuning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。