高可塑性组件更利于ViT微调，打破平滑性越好的假设。
Vision Transformer Finetuning Benefits from Non-Smooth Components
- 用可塑性衡量组件对输入变化的响应灵敏度
- 实验显示注意力层和前馈层高可塑性提升微调效果
- 适合关注模型迁移性能的开发者参考
Transformer架构的平滑性在泛化、训练稳定性和对抗鲁棒性方面已有广泛研究，但在迁移学习中的作用仍不明确。本文分析了视觉Transformer组件适应输入变化的能力，即其可塑性——定义为平均变化率，反映对输入扰动的敏感程度，高可塑性意味着低平滑性。通过理论分析与超过1000次大规模视觉Transformer微调实验，结果表明该视角能为适配过程提供合理指导。关键发现是：注意力模块和前馈层的高可塑性始终带来更好的微调表现。这一发现挑战了‘平滑性有益’的普遍认知，为理解Transformer功能特性提供了新视角。代码已公开于https://github.com/ambroiseodt/vit-plasticity。
原文摘要 · Abstract (English)
The smoothness of the transformer architecture has been extensively studied in the context of generalization, training stability, and adversarial robustness. However, its role in transfer learning remains poorly understood. In this paper, we analyze the ability of vision transformer components to adapt their outputs to changes in inputs, or, in other words, their \emph{plasticity}. Defined as an average rate of change, it captures the sensitivity to input perturbation; in particular, a high plasticity implies a low smoothness. Our theoretical analysis and extensive experiments -- over $1,000$ finetuning runs on large-scale vision transformers -- showcase that this perspective provides principled guidance in choosing the components to prioritize during adaptation. A key takeaway for practitioners is that the high plasticity of the attention modules and feedforward layers consistently leads to better finetuning performance. Our findings depart from the prevailing assumption that smoothness is desirable, offering a novel perspective on transformers' functional properties. The code is available at https://github.com/ambroiseodt/vit-plasticity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。