arXiv:2605.10161cs.LG2026-05

根据激活模式动态调整卷积层权重衰减,提升模型泛化能力

OUIDecay: Adaptive Layer-wise Weight Decay for CNNs Using Online Activation Patterns

  • 基于激活模式实时监测各层结构行为,动态调节权重衰减
  • 在8个实验中7个达到最佳验证损失,平均表现优于固定衰减
  • 无需验证集和梯度信息,轻量级适合在线训练

权重衰减是训练卷积神经网络最常用的正则化方法之一,但通常以固定系数应用于所有层。本文提出OUIDecay,一种基于过拟合-欠拟合指示器(OUI)的自适应分层时变权重衰减调度器。OUIDecay通过轻量级批处理形式的OUI在线监控每层的结构行为,并周期性地调整其权重衰减相对于网络其他层的比例。与依赖梯度的方法不同,该方法基于激活模式提取功能信息,无需验证数据。在EfficientNet-B0+Stanford Cars、ResNet50+Food101、DenseNet121+CIFAR100、MobileNetV2+CIFAR10上的实验表明,OUIDecay在8个评估设置中7个达到最低均值验证损失,证明激活驱动的自适应衰减是一种有效且实用的替代方案,同时保持轻量与在线适用性。

原文摘要 · Abstract (English)

Weight decay remains one of the most widely used regularization mechanisms for training convolutional neural networks, yet it is still commonly applied as a fixed coefficient shared by all layers throughout training. This uniform treatment ignores that different layers may follow different structural dynamics and therefore may require different regularization strengths. In this work, we propose OUIDecay, an adaptive layer-wise and time-dependent weight decay scheduler for CNNs driven by the Overfitting-Underfitting Indicator (OUI), an activation-based metric previously shown to provide early information about regularization quality. OUIDecay uses a lightweight batch-based formulation of OUI to monitor the structural behavior of each layer online and periodically rescales its weight decay relative to the other layers in the network. Unlike gradient-based adaptive decay methods, our approach relies on functional information extracted from activation patterns and does not require validation data. Experiments on EfficientNet-B0 with Stanford Cars, ResNet50 with Food101, DenseNet121 with CIFAR100, and MobileNetV2 with CIFAR10 show that OUIDecay achieves the best mean best-validation-loss in 7 out of 8 evaluated settings. These results indicate that activation-driven weight decay adaptation is a practical and effective alternative to fixed decay and gradient-based adaptive decay, while keeping the method lightweight and suitable for online use.

权重衰减自适应CNN在线学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。