用注意力机制增强特征适配，让图像分类更精准
A Contrastive Learning Framework Empowered by Attention-based Feature Adaptation for Street-View Image Classification
- 在CLIP基础上加入多头注意力模块，捕捉局部图像依赖关系
- 仅140万参数，在8个任务上达到新最佳性能
- 适合需要轻量高效图像分类的自动驾驶等场景
街景图像属性分类是图像分类的重要下游任务，广泛应用于自动驾驶、城市分析和高精地图构建。无论从头训练、使用预训练权重还是微调大模型，该任务仍计算成本高昂。尽管像CLIP这样的预训练视觉语言模型能提供丰富图像表征，但现有适配或微调方法通常依赖全局图像嵌入，难以捕捉复杂、杂乱街景中的细粒度局部属性。为此，我们提出CLIP-MHAdapter，一种轻量级CLIP适配变体，在瓶颈MLP中引入对补丁令牌操作的多头自注意力机制，以建模补丁间依赖关系。该模型仅含约140万可训练参数，在Global StreetScapes数据集的8项属性分类任务上表现优异或领先，取得新的最佳结果，同时保持低计算开销。代码已公开于https://github.com/SpaceTimeLab/CLIP-MHAdapter。
原文摘要 · Abstract (English)
Street-view image attribute classification is a vital downstream task of image classification, enabling applications such as autonomous driving, urban analytics, and high-definition map construction. It remains computationally demanding whether training from scratch, initialising from pre-trained weights, or fine-tuning large models. Although pre-trained vision-language models such as CLIP offer rich image representations, existing adaptation or fine-tuning methods often rely on their global image embeddings, limiting their ability to capture fine-grained, localised attributes essential in complex, cluttered street scenes. To address this, we propose CLIP-MHAdapter, a variant of the current lightweight CLIP adaptation paradigm that appends a bottleneck MLP equipped with multi-head self-attention operating on patch tokens to model inter-patch dependencies. With approximately 1.4 million trainable parameters, CLIP-MHAdapter achieves superior or competitive accuracy across eight attribute classification tasks on the Global StreetScapes dataset, attaining new state-of-the-art results while maintaining low computational cost. The code is available at https://github.com/SpaceTimeLab/CLIP-MHAdapter.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。