用视觉引导动态调整提示词,提升小样本细粒度分类准确率
Adaptive Prompt Tuning: Vision Guided Prompt Tuning with Cross-Attention for Fine-Grained Few-Shot Learning
- 通过交叉注意力实时调整文本提示,匹配图像特征
- 在CUBirds等数据集上显著超越静态提示方法
- 结合蒙特卡洛丢弃提升预测可信度,适合高风险场景
小样本细粒度分类因需区分细微类别差异且数据稀缺而极具挑战。本文提出一种新方法,通过视觉引导的自适应提示调优增强CLIP模型。与传统静态提示或依赖视觉标记的方法不同,该方法利用交叉注意力机制,动态优化当前图像对应的文本提示,实现文本特征与视觉变压器提取图像块的精准对齐,尤其适用于类内差异大、类间差异小的数据集。在CUBirds、Oxford Flowers和FGVC Aircraft等数据集上的实验表明,该方法显著优于静态提示调优。为确保预测可靠性,引入蒙特卡洛丢弃技术,提升模型预测置信度与不确定性估计能力,帮助识别可信与需验证的预测结果。此动态策略推动了小样本细粒度分类的性能边界。
原文摘要 · Abstract (English)
Few-shot, fine-grained classification in computer vision poses significant challenges due to the need to differentiate subtle class distinctions with limited data. This paper presents a novel method that enhances the Contrastive Language-Image Pre-Training (CLIP) model through adaptive prompt tuning, guided by real-time visual inputs. Unlike existing techniques such as Context Optimization (CoOp) and Visual Prompt Tuning (VPT), which are constrained by static prompts or visual token reliance, the proposed approach leverages a cross-attention mechanism to dynamically refine text prompts for the image at hand. This enables an image-specific alignment of textual features with image patches extracted from the Vision Transformer, making the model more effective for datasets with high intra-class variance and low inter-class differences. The method is evaluated on several datasets, including CUBirds, Oxford Flowers, and FGVC Aircraft, showing significant performance gains over static prompt tuning approaches. To ensure these performance gains translate into trustworthy predictions, we integrate Monte-Carlo Dropout in our approach to improve the reliability of the model predictions and uncertainty estimates. This integration provides valuable insights into the model's predictive confidence, helping to identify when predictions can be trusted and when additional verification is necessary. This dynamic approach offers a robust solution, advancing the state-of-the-art for few-shot fine-grained classification.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。