解决音频模型提示学习中旧类好新类差的问题
ZEBRA: Zero-Shot Entropy-Regularized Prompt Learning for Base-to-Novel Generalization in Audio-Language Models
- 用零样本与提示学习结果融合，避免过拟合旧类别
- 引入自熵正则化，提升对新类别的泛化能力
- 适合需要兼顾旧新类别性能的音频分类任务
音频-语言模型（ALMs）通过对齐音频与文本类别描述，实现强大的零样本性能。尽管提示学习能通过少量有标签数据提升基类准确率，但我们发现存在关键矛盾：它常导致新类别性能下降，甚至低于零样本表现，暴露出提示学习在基类到新类泛化上的差距。为此，我们提出ZEBRA（零样本熵正则化提示学习），一种即插即用框架，将零样本输出与提示学习输出融合，并采用自熵正则化抑制对基类的过拟合。在多个音频分类数据集上的实验表明，ZEBRA在保持强基类性能的同时显著提升新类别表现，大幅缩小了基类到新类的性能差距。代码已开源。
原文摘要 · Abstract (English)
Audio-Language Models (ALMs) achieve strong zero-shot performance by aligning audio with textual class descriptions. Although prompt learning improves accuracy on base classes through few-shot supervised adaptation, we observe a critical trade-off: it often degrades performance on novel classes, sometimes falling below zero-shot accuracy. This exposes a base-to-novel generalization gap in prompt learning for ALMs. To address this issue, we propose \textbf{ZEBRA} (Zero-shot Entropy-Regularized Prompt Learning for Base-to-Novel Generalization), a plug-and-play framework that fuses zero-shot logits with prompt-learning logits, and employs self-entropy regularization to reduce overfitting to base classes. Experiments across multiple audio classification datasets show that ZEBRA consistently improves novel-class performance while maintaining strong base accuracy, significantly reducing the base-to-novel gap compared to standard prompt learning. The code is available at: https://github.com/asif-hanif/zebra.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。