arXiv:2606.25256cs.CVcs.LG2026-06

用输入数据预初始化卷积层,零成本提升模型性能。

Pre-Warm: Input-Conditioned Weight Initialization for Convolutional Neural Networks

  • 从单批数据提取局部块,聚类后生成滤波器初始值
  • 在5个基准数据集上显著提升准确率,最高8/8胜出
  • 无需修改结构,几行代码即可接入现有训练流程

我们提出Pre-Warm,一种无需训练开销的数据依赖式卷积神经网络首层权重初始化方法。在首次前向传播前,Pre-Warm从单个训练批次中提取均值中心化局部块,使用MiniBatchKMeans进行聚类,应用逆曼哈顿空间加权,并利用得到的质心初始化一半首层滤波器(另一半保留Kaiming初始化)。除一个对结果不敏感的尺度参数外,其余超参数均有闭式规则;尺度参数由块维度推导出的Kaiming对齐约束确定。对于灰度图像,采用Otsu前景密度;对于自然彩色图像,使用均值中心化块的均值L2范数。两种规则均能准确预测网格搜索中观察到的最佳块数。在五个标准基准数据集——MNIST、Fashion-MNIST、CIFAR-10、SVHN和CIFAR-100——及8次种子重复实验中,Pre-Warm相对于标准Kaiming初始化均取得统计显著的精度提升(所有数据集p < 0.05,SVHN为p = 0.0007且8/8胜出,CIFAR-100为p = 0.0033且7/8胜出)。该方法开销极低,无需架构改动,仅需少量代码即可融入现有训练流程。Pre-Warm表明,即使轻量级输入相关信号也能显著改善现代卷积网络的优化轨迹。

原文摘要 · Abstract (English)

We introduce Pre-Warm, a simple yet effective zero-training-cost method for data-conditioned initialization of the first convolutional layer. Before the first forward pass, Pre-Warm extracts mean-centered local patches from a single training batch, clusters them with MiniBatchKMeans, applies inverse Manhattan spatial weighting, and uses the resulting centroids to initialize half of the first-layer filters (the remainder retain Kaiming initialization). We derive closed-form rules for all hyperparameters except a single insensitive scale parameter, though we derive a Kaiming parity bound on scale from patch dimensionality. For grayscale datasets we use Otsu's foreground density; for natural color images we use the mean L2 norm of mean-centered patches. Both rules accurately predict the optimal patch count observed in grid search. Across five standard benchmarks -- MNIST, Fashion-MNIST, CIFAR-10, SVHN, and CIFAR-100 -- and 8-seed paired experiments, Pre-Warm yields statistically significant accuracy improvements over standard Kaiming initialization (p < 0.05 on all datasets, p = 0.0007 on SVHN with 8/8 wins, p = 0.0033 on CIFAR-100 with 7/8 wins). The method adds negligible overhead, requires no architectural changes, and integrates into existing training pipelines with only a few lines of code. Pre-Warm demonstrates that even a lightweight, input-dependent signal can meaningfully improve optimization trajectories in modern convolutional networks.

卷积网络权重初始化零训练成本数据驱动

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。