提出自适应不对称适配器，缓解视觉语言模型少样本学习中的分支偏差问题。
A$_3$B$_2$: Adaptive Asymmetric Adapter for Alleviating Branch Bias in Vision-Language Image Classification with Few-Shot Learning
- 设计动态抑制机制，根据预测不确定性自动控制图像分支适配强度。
- 在11个数据集上优于11种主流基线方法，尤其在分布外场景表现更优。
- 轻量级架构适合资源受限场景，无需人工调参，适合少样本视觉任务应用。
大规模视觉-语言模型（如CLIP）的高效迁移学习方法实现了强大的少样本迁移能力，但现有适配方法采用固定微调范式，隐含假设图像与文本分支重要性均等，这在图像分类中尚未系统研究。通过大量分析，我们发现视觉-语言图像分类存在分支偏差问题：在分布外设置下，仅适配图像编码器并不总能提升性能。受此启发，我们提出A₃B₂——一种自适应不对称适配器，用于缓解少样本学习中的分支偏差。A₃B₂引入不确定性感知适配衰减（UAAD）机制，在预测不确定性高时自动抑制图像分支的适配，实现无需人工干预的数据驱动控制。其架构借鉴专家混合模型并结合负载均衡正则化，具有轻量级不对称设计。在三个少样本图像分类任务、11个数据集上的广泛实验表明，A₃B₂始终优于11种竞争性的提示与适配器基线方法。
原文摘要 · Abstract (English)
Efficient transfer learning methods for large-scale vision-language models ($e.g.$, CLIP) enable strong few-shot transfer, yet existing adaptation methods follow a fixed fine-tuning paradigm that implicitly assumes a uniform importance of the image and text branches, which has not been systematically studied in image classification. Through extensive analysis, we reveal a Branch Bias issue in vision-language image classification: adapting the image encoder does not always improve performance under out-of-distribution settings. Motivated by this observation, we propose A$_3$B$_2$, an Adaptive Asymmetric Adapter that alleviates Branch Bias in few-shot learning. A$_3$B$_2$ introduces Uncertainty-Aware Adapter Dampening (UAAD), which automatically suppresses image-branch adaptation when prediction uncertainty is high, enabling soft and data-driven control without manual intervention. Architecturally, A$_3$B$_2$ adopts a lightweight asymmetric design inspired by mixture-of-experts with Load Balancing Regularization. Extensive experiments on three few-shot image classification tasks across 11 datasets demonstrate that A$_3$B$_2$ consistently outperforms 11 competitive prompt- and adapter-based baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。