arXiv:2509.12768cs.CVcs.LG2025-09中稿 · publication at the…

提出双阶段自适应标记精炼方法,提升视觉Transformer在少样本下的分类性能。

BATR-FST: Bi-Level Adaptive Token Refinement for Few-Shot Transformers

  • 分两阶段优化:预训练用掩码图像建模,微调用聚类与不确定性加权精炼标记
  • 在3个基准数据集上1-shot和5-shot均显著超越现有方法
  • 适合少样本学习、视觉Transformer改进及需要强先验的任务

视觉变压器(ViTs)在计算机视觉中展现出巨大潜力,但在少样本学习中受限于标记级交互优化困难、训练数据有限以及缺乏强归纳偏置。现有方法多依赖固定匹配或简单相似度度量,难以有效融合全局上下文与局部特征精炼。为此,我们提出双层自适应标记精炼框架(BATR-FST),通过两阶段策略逐步提升标记表示并保持稳健的归纳偏置。预训练阶段采用掩码图像建模(MIM)重建被遮蔽图像区域,生成可迁移的块级表征。元微调阶段引入双层自适应标记精炼模块:利用标记聚类捕捉局部交互,不确定性感知标记加权优先保留可靠特征,双层注意力平衡簇内与簇间关系,实现深度标记精炼;同时,图标记传播确保支持与查询实例间的语义一致性,类别分离惩罚项维持类间边界,增强判别能力。在三个少样本基准数据集上的大量实验表明,BATR-FST在1-shot和5-shot场景下均取得更优性能,显著提升基于Transformer的少样本分类效果。

原文摘要 · Abstract (English)

Vision Transformers (ViTs) have shown significant promise in computer vision applications. However, their performance in few-shot learning is limited by challenges in refining token-level interactions, struggling with limited training data, and developing a strong inductive bias. Existing methods often depend on inflexible token matching or basic similarity measures, which limit the effective incorporation of global context and localized feature refinement. To address these challenges, we propose Bi-Level Adaptive Token Refinement for Few-Shot Transformers (BATR-FST), a two-stage approach that progressively improves token representations and maintains a robust inductive bias for few-shot classification. During the pre-training phase, Masked Image Modeling (MIM) provides Vision Transformers (ViTs) with transferable patch-level representations by recreating masked image regions, providing a robust basis for subsequent adaptation. In the meta-fine-tuning phase, BATR-FST incorporates a Bi-Level Adaptive Token Refinement module that utilizes Token Clustering to capture localized interactions, Uncertainty-Aware Token Weighting to prioritize dependable features, and a Bi-Level Attention mechanism to balance intra-cluster and inter-cluster relationships, thereby facilitating thorough token refinement. Furthermore, Graph Token Propagation ensures semantic consistency between support and query instances, while a Class Separation Penalty preserves different class borders, enhancing discriminative capability. Extensive experiments on three benchmark few-shot datasets demonstrate that BATR-FST achieves superior results in both 1-shot and 5-shot scenarios and improves the few-shot classification via transformers.

少样本学习视觉Transformer标记精炼自适应

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。