arXiv:2508.07577cs.CVcs.LG2025-08被引 3

通过分析归一化参数变化,提升小样本下视觉Transformer的迁移性能。

Exploiting Layer Normalization Fine-tuning in Visual Transformer Foundation Models for Classification

  • 基于细调后层归一化参数偏移量设计可调节缩放因子λ。
  • 在少样本和域外数据上,性能提升显著,尤其在低代表性的目标数据中。
  • 适合处理数据稀缺或领域分布差异大的图像分类任务。

层归一化(LayerNorm)在视觉变换器(ViTs)中至关重要,但在数据稀缺和领域偏移下的微调动态仍缺乏研究。本文发现,微调后的层归一化参数偏移(即层归一化偏移)可反映源域与目标域之间的转换;其有效性取决于目标训练样本对目标域的代表性程度,我们提出了微调偏移比(FSR)进行量化。基于此,我们提出一种简单有效的缩放机制,使用与FSR负相关的标量λ,将学习到的层归一化偏移对齐至全代表性数据下的理想偏移,并结合循环框架进一步优化层归一化微调。在自然图像与病理图像上的大量实验验证了该框架的有效性,涵盖分布内(ID)与分布外(OOD)设置及不同目标样本规模。值得注意的是,OOD任务的FSR较低,λ较高,尤其是在数据稀疏时,表明目标训练样本代表性不足。此外,病理数据微调的ViT表现更接近于ID设置,偏好保守的层归一化更新。研究揭示了层归一化在迁移学习中的未被充分探索的动态,并提供了实用的微调策略。

原文摘要 · Abstract (English)

LayerNorm is pivotal in Vision Transformers (ViTs), yet its fine-tuning dynamics under data scarcity and domain shifts remain underexplored. This paper shows that shifts in LayerNorm parameters after fine-tuning (LayerNorm shifts) are indicative of the transitions between source and target domains; its efficacy is contingent upon the degree to which the target training samples accurately represent the target domain, as quantified by our proposed Fine-tuning Shift Ratio ($FSR$). Building on this, we propose a simple yet effective rescaling mechanism using a scalar $λ$ that is negatively correlated to $FSR$ to align learned LayerNorm shifts with those ideal shifts achieved under fully representative data, combined with a cyclic framework that further enhances the LayerNorm fine-tuning. Extensive experiments across natural and pathological images, in both in-distribution (ID) and out-of-distribution (OOD) settings, and various target training sample regimes validate our framework. Notably, OOD tasks tend to yield lower $FSR$ and higher $λ$ in comparison to ID cases, especially with scarce data, indicating under-represented target training samples. Moreover, ViTFs fine-tuned on pathological data behave more like ID settings, favoring conservative LayerNorm updates. Our findings illuminate the underexplored dynamics of LayerNorm in transfer learning and provide practical strategies for LayerNorm fine-tuning.

视觉Transformer层归一化少样本学习迁移学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。