arXiv:2507.10283cs.CV2025-07被引 1

用语义聚类动态生成图像令牌,提升分类精度

FTCFormer: Fuzzy Token Clustering Transformer for Image Classification

  • 根据语义重要性动态聚类生成视觉令牌,非固定网格划分
  • 在5个细粒度数据集上提升1.43%,多领域平均增益超0.5%
  • 适合需要精准特征提取的图像分类任务,尤其医疗与遥感

基于Transformer的深度神经网络在多种计算机视觉任务中取得显著成功,主要得益于其长程自注意力机制和可扩展性。然而,大多数Transformer架构将图像均匀划分为网格状视觉令牌,忽略了图像区域的潜在语义信息,导致特征表示不佳。为此,我们提出模糊令牌聚类Transformer(FTCFormer),引入一种新型基于聚类的下采样模块,根据语义意义而非空间位置动态生成视觉令牌。该方法对信息量少的区域分配较少令牌,对语义重要的区域分配更多令牌,不受空间邻近性或形状不规则性影响。为进一步增强特征提取与表示,我们提出密度峰值聚类-模糊K近邻(DPC-FKNN)机制确定聚类中心,使用空间连通性评分(SCS)进行令牌分配,并采用通道级合并(Cmerge)策略完成令牌融合。在32个跨领域数据集上的大量实验验证了FTCFormer的有效性,在图像分类任务中持续优于TCFormer基线,五细粒度数据集提升1.43%,六自然图像数据集提升1.09%,三医学数据集提升0.97%,四遥感数据集提升0.55%。代码已开源:https://github.com/BaoBao0926/FTCFormer/tree/main。

原文摘要 · Abstract (English)

Transformer-based deep neural networks have achieved remarkable success across various computer vision tasks, largely attributed to their long-range self-attention mechanism and scalability. However, most transformer architectures embed images into uniform, grid-based vision tokens, neglecting the underlying semantic meanings of image regions, resulting in suboptimal feature representations. To address this issue, we propose Fuzzy Token Clustering Transformer (FTCFormer), which incorporates a novel clustering-based downsampling module to dynamically generate vision tokens based on the semantic meanings instead of spatial positions. It allocates fewer tokens to less informative regions and more to represent semantically important regions, regardless of their spatial adjacency or shape irregularity. To further enhance feature extraction and representation, we propose a Density Peak Clustering-Fuzzy K-Nearest Neighbor (DPC-FKNN) mechanism for clustering center determination, a Spatial Connectivity Score (SCS) for token assignment, and a channel-wise merging (Cmerge) strategy for token merging. Extensive experiments on 32 datasets across diverse domains validate the effectiveness of FTCFormer on image classification, showing consistent improvements over the TCFormer baseline, achieving gains of improving 1.43% on five fine-grained datasets, 1.09% on six natural image datasets, 0.97% on three medical datasets and 0.55% on four remote sensing datasets. The code is available at: https://github.com/BaoBao0926/FTCFormer/tree/main.

图像分类Transformer聚类动态令牌

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。