将ViT与自组织映射结合，提升小数据下的视觉模型性能
Simple Self Organizing Map with Vision Transformers
- 用SOM结构约束ViT特征空间，增强小数据下的表征能力
- 在小规模数据集上，自监督与监督任务均显著提升准确率
- 适合研究小样本学习、无监督表示的学者参考
视觉变换器（ViTs）在多种视觉任务中表现卓越，但在小数据集上因缺乏归纳偏置而表现不佳。现有方法通常通过预训练任务或从卷积神经网络（CNN）蒸馏知识来弥补这一缺陷。相比之下，自组织映射（SOM）作为一种广泛采用的自监督框架，天然具备保持拓扑与空间结构的能力，是直接解决ViTs在小数据场景下局限性的理想候选。尽管如此，将SOM与现代深度学习架构结合的研究仍几乎空白。本文首次探索了视觉变换器（ViTs）与自组织映射（SOMs）之间的协同潜力，旨在填补这一关键研究空白。结果表明，两者可相互增强，在自监督与监督任务中均实现显著性能提升。代码已公开于GitHub。
原文摘要 · Abstract (English)
Vision Transformers (ViTs) have demonstrated exceptional performance in various vision tasks. However, they tend to underperform on smaller datasets due to their inherent lack of inductive biases. Current approaches address this limitation implicitly-often by pairing ViTs with pretext tasks or by distilling knowledge from convolutional neural networks (CNNs) to strengthen the prior. In contrast, Self-Organizing Maps (SOMs), a widely adopted self-supervised framework, are inherently structured to preserve topology and spatial organization, making them a promising candidate to directly address the limitations of ViTs in limited or small training datasets. Despite this potential, equipping SOMs with modern deep learning architectures remains largely unexplored. In this study, we conduct a novel exploration on how Vision Transformers (ViTs) and Self-Organizing Maps (SOMs) can empower each other, aiming to bridge this critical research gap. Our findings demonstrate that these architectures can synergistically enhance each other, leading to significantly improved performance in both unsupervised and supervised tasks. Code is publicly available on GitHub.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。