用金字塔提示和邻近选择,让模型在少样本下更准识别新类别。
KNN Transformer with Pyramid Prompts for Few-Shot Learning
- 通过选最相关token做注意力,逐步过滤无关信息。
- 多尺度文本提示增强视觉特征,提升对空间变化的鲁棒性。
- 适合少样本场景下的图像分类,尤其对噪声敏感任务有效。
少样本学习(FSL)旨在仅用少量标注数据识别新类别。现有方法尝试用文本提示调制视觉特征,但难以捕捉文本与视觉特征间的复杂语义关系。同时,原始自注意力易受图像中无关信息干扰,导致语义先验效果受限。为此,提出K-NN Transformer与金字塔提示(KTPP):首先,基于K个最近邻的上下文注意力(KCA)仅选择最相关的K个标记计算注意力矩阵,并引入所有标记均值作为全局上下文提示,在三级级联中逐步抑制无关标记;其次,引入金字塔提示(PCP),通过文本类提示与多尺度视觉特征交互,使ViT能根据不同尺度的丰富语义动态调整视觉特征权重,增强对空间变化的鲁棒性;最后,通过KCA将增强的视觉特征与类感知提示交互,提取类特定特征。该方法通过深度跨模态交互,显著提升少样本场景下的无噪视觉表示能力。在四个基准数据集上的大量实验验证了其有效性。
原文摘要 · Abstract (English)
Few-Shot Learning (FSL) aims to recognize new classes with limited labeled data. Recent studies have attempted to address the challenge of rare samples with textual prompts to modulate visual features. However, they usually struggle to capture complex semantic relationships between textual and visual features. Moreover, vanilla self-attention is heavily affected by useless information in images, severely constraining the potential of semantic priors in FSL due to the confusion of numerous irrelevant tokens during interaction. To address these aforementioned issues, a K-NN Transformer with Pyramid Prompts (KTPP) is proposed to select discriminative information with K-NN Context Attention (KCA) and adaptively modulate visual features with Pyramid Cross-modal Prompts (PCP). First, for each token, the KCA only selects the K most relevant tokens to compute the self-attention matrix and incorporates the mean of all tokens as the context prompt to provide the global context in three cascaded stages. As a result, irrelevant tokens can be progressively suppressed. Secondly, pyramid prompts are introduced in the PCP to emphasize visual features via interactions between text-based class-aware prompts and multi-scale visual features. This allows the ViT to dynamically adjust the importance weights of visual features based on rich semantic information at different scales, making models robust to spatial variations. Finally, augmented visual features and class-aware prompts are interacted via the KCA to extract class-specific features. Consequently, our model further enhances noise-free visual representations via deep cross-modal interactions, extracting generalized visual representation in scenarios with few labeled samples. Extensive experiments on four benchmark datasets demonstrate the effectiveness of our method.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。