让音频模型的注意力头按类别分配权重，提升分类精度。
CALM: Class-Conditional Sparse Attention Vectors for Large Audio-Language Models
- 为每个类别学习注意力头的重要性权重，实现差异化贡献。
- 在音频分类等任务上最高提升14.52%准确率。
- 适合需要高精度分类的少样本音频任务研究者。
大型音频-语言模型（LALMs）在零样本音频问答和抽象推理任务中表现优异，但在某些判别性任务（如音频分类）上仍落后于专用模型。近期研究表明，从LALM中选取稀疏注意力头并用简单投票可作为强判别特征提取器。但现有方法对所有选中头赋予相同权重，隐含假设各头在所有语义类别中贡献均等。本文提出类条件稀疏注意力向量（CALM），一种少样本分类方法，通过学习类别依赖的重要性权重，使不同注意力头能专精于特定语义类别，并按估计可靠性贡献预测。在多个少样本音频与音视频分类基准上，该方法相比最优均匀投票方法，音频分类、音视频分类、欺骗检测分别提升14.52%、1.53%、8.35%绝对准确率。
原文摘要 · Abstract (English)
Large audio-language models (LALMs) exhibit strong zero-shot capabilities in multiple downstream tasks, such as audio question answering (AQA) and abstract reasoning; however, these models still lag behind specialized models for certain discriminative tasks (e.g., audio classification). Recent studies show that sparse subsets of attention heads within an LALM can serve as strong discriminative feature extractors for downstream tasks such as classification via simple voting schemes. However, these methods assign uniform weights to all selected heads, implicitly assuming that each head contributes equally across all semantic categories. In this work, we propose Class-Conditional Sparse Attention Vectors for Large Audio-Language Models, a few-shot classification method that learns class-dependent importance weights over attention heads. This formulation allows individual heads to specialize in distinct semantic categories and to contribute to ensemble predictions proportionally to their estimated reliability. Experiments on multiple few-shot audio and audiovisual classification benchmarks and tasks demonstrate that our method consistently outperforms state-of-the-art uniform voting-based approaches by up to 14.52%, 1.53%, 8.35% absolute gains for audio classification, audio-visual classification, and spoofing detection respectively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。