通过观察预训练权重结构，为MLP设计简单初始化方法，提升小规模图像任务训练速度。
Mimetic Initialization of MLPs
- 借鉴预训练模型权重结构，设计仅需首层均值非零的MLP初始化方法。
- 在CIFAR-10和ImageNet-1k上显著加快训练速度，效果虽小于空间混合层但可叠加。
- 适合希望优化Transformer或CNN中MLP模块训练效率的研究者。
模拟初始化（Mimetic initialization）利用预训练模型作为良好初始化的案例，通过观察训练后权重中的结构特征，启发新的简单初始化方法。此前该方法仅应用于空间混合层，如卷积、自注意力和状态空间层。本文首次将其拓展至通道混合层，即多层感知机（MLPs）。我们提出一种极简方法：仅使第一层权重均值非零。该方法在小规模视觉任务（如CIFAR-10和ImageNet-1k）上显著加速训练。尽管其增益远小于空间混合层的初始化效果，但可与之结合使用，产生额外正向影响。
原文摘要 · Abstract (English)
Mimetic initialization uses pretrained models as case studies of good initialization, using observations of structures in trained weights to inspire new, simple initialization techniques. So far, it has been applied only to spatial mixing layers, such convolutional, self-attention, and state space layers. In this work, we present the first attempt to apply the method to channel mixing layers, namely multilayer perceptrons (MLPs). Our extremely simple technique for MLPs -- to give the first layer a nonzero mean -- speeds up training on small-scale vision tasks like CIFAR-10 and ImageNet-1k. Though its effect is much smaller than spatial mixing initializations, it can be used in conjunction with them for an additional positive effect.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。