提升视觉语言模型在分布偏移下的泛化能力,解决类别稀疏和语义混淆问题。
Class-Aware Prototype Learning with Negative Contrast for Test-Time Adaptation of Vision-Language Models
- 动态调整每类原型容量,结合激活历史与频率,保留罕见类别知识
- 引入负样本对比学习,增强类别间区分度,提升分类性能
- 轻量级框架,仅更新文本原型,适合部署时快速自适应
视觉语言模型(VLMs)通过大规模图文预训练展现出强大的零样本泛化能力,但在部署分布与训练分布不一致时性能下降。为应对这一问题,测试时自适应(TTA)方法利用未标注目标数据更新模型。然而现有方法常忽略两个关键挑战:长尾分布下的原型退化,以及语义相近类别的混淆。为此,本文提出类感知原型学习与负对比(CPL-NC),一种专为VLM设计的轻量级TTA框架,以增强分布偏移下的泛化能力。CPL-NC引入类感知原型缓存模块,根据测试时频率和激活历史动态调整每类容量,并通过再生机制恢复不活跃类别的知识。同时,负对比学习机制识别并约束困难的视觉-文本负样本,提升类别可分性。该框架采用非对称优化策略,仅更新文本原型,锚定于稳定的视觉特征。在15个基准测试上,无论使用ResNet-50还是ViT-B/16骨干网络,CPL-NC均持续优于现有TTA方法。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) demonstrate impressive zero-shot generalization through large-scale image-text pretraining, yet their performance can drop once the deployment distribution diverges from the training distribution. To address this, Test-Time Adaptation (TTA) methods update models using unlabeled target data. However, existing approaches often ignore two key challenges: prototype degradation in long-tailed distributions and confusion between semantically similar classes. To tackle these issues, we propose \textbf{C}lass-Aware \textbf{P}rototype \textbf{L}earning with \textbf{N}egative \textbf{C}ontrast(\textbf{CPL-NC}), a lightweight TTA framework designed specifically for VLMs to enhance generalization under distribution shifts. CPL-NC introduces a \textit{Class-Aware Prototype Cache} Module that dynamically adjusts per-class capacity based on test-time frequency and activation history, with a rejuvenation mechanism for inactive classes to retain rare-category knowledge. Additionally, a \textit{Negative Contrastive Learning} Mechanism identifies and constrains hard visual-textual negatives to improve class separability. The framework employs asymmetric optimization, refining only textual prototypes while anchoring on stable visual features. Experiments on 15 benchmarks show that CPL-NC consistently outperforms prior TTA methods across both ResNet-50 and ViT-B/16 backbones.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。