训练方式决定视觉模型哪些层关键，影响模型决策效率。
How Do Training Methods Influence the Utilization of Vision Models?
- 对比不同训练流程对各层重要性的影响。
- 自监督训练提升浅层重要性，深层利用率低。
- 对抗训练则相反，适合关注深层特征的场景。
并非所有可学习参数都同等影响神经网络的决策函数，部分层权重重置为随机值仍不影响模型输出。我们重新审视了架构与任务复杂度对这一现象的影响，并提出：训练方法是否也会影响该现象？在保持架构和训练数据一致的前提下，我们在 ImageNet-1k 分类任务上测试了多种训练流程。结果表明，训练方法显著影响各层对决策的重要性。例如，改进型训练和自监督训练使早期层更加关键，但严重低估深层参数的作用；而对抗训练则呈现相反趋势。这些初步发现拓展了对神经网络内部机制的理解。
原文摘要 · Abstract (English)
Not all learnable parameters (e.g., weights) contribute equally to a neural network's decision function. In fact, entire layers' parameters can sometimes be reset to random values with little to no impact on the model's decisions. We revisit earlier studies that examined how architecture and task complexity influence this phenomenon and ask: is this phenomenon also affected by how we train the model? We conducted experimental evaluations on a diverse set of ImageNet-1k classification models to explore this, keeping the architecture and training data constant but varying the training pipeline. Our findings reveal that the training method strongly influences which layers become critical to the decision function for a given task. For example, improved training regimes and self-supervised training increase the importance of early layers while significantly under-utilizing deeper layers. In contrast, methods such as adversarial training display an opposite trend. Our preliminary results extend previous findings, offering a more nuanced understanding of the inner mechanics of neural networks. Code: https://github.com/paulgavrikov/layer_criticality
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。