通过调控训练初期的Mixup策略,显著提升DNN模型性能。
Enlightenment Period Improving DNN Performance
- 在训练初期(准确率50%前)引入Mixup会引发梯度干扰和激活恢复双效应。
- 小规模数据时暂停Mixup,大规模数据时提升Mixup强度可优化效果。
- 适用于视觉、语言等多种任务,尤其对小型模型和欠拟合场景有效。
深度神经网络训练初期存在一个关键的短暂阶段,即从开始到准确率达到约50%的时期,我们称之为启蒙期。此阶段内,无序表征迅速向有序结构转变。基于相变理论建模与实验验证,我们发现在此期间应用Mixup数据增强具有双重影响:一方面产生梯度干扰效应,阻碍性能提升;另一方面带来激活恢复效应,帮助饱和神经元重新获得梯度更新。进一步发现,随着样本量或模型参数量增加,负向干扰减弱,两效应平衡发生偏移。据此提出三种仅调整早期数据分布的策略:小规模场景用混合暂停策略,大规模且欠拟合时用α增强策略,不适合使用Mixup的任务(如时间序列、大语言模型)则采用高损失样本剔除策略。大量实验表明,这些方法在ViT、ResNet等架构上于CIFAR、ImageNet-1K等数据集均表现优异。该研究为提升模型性能提供了基于训练早期动态的全新视角。代码已公开。
原文摘要 · Abstract (English)
The start of deep neural network training is characterized by a brief yet critical phase that lasts from the beginning of the training until the accuracy reaches approximately 50\%. During this phase, disordered representations rapidly transition toward ordered structure, and we term this phase the Enlightenment Period. Through theoretical modeling based on phase transition theory and experimental validation, we reveal that applying Mixup data augmentation during this phase has a dual effect: it introduces a Gradient Interference Effect that hinders performance, while also providing a beneficial Activation Revival Effect to restore gradient updates for saturated neurons. We further demonstrate that this negative interference diminishes as the sample set size or the model parameter size increases, thereby shifting the balance between these two effects. Based on these findings, we propose three strategies that improve performance by solely adjusting the training data distribution within this brief period: the Mixup Pause Strategy for small-scale scenarios, the Alpha Boost Strategy for large-scale scenarios with underfitting, and the High-Loss Removal Strategy for tasks where Mixup is inapplicable (e.g., time series and large language models). Extensive experiments show that these strategies achieve superior performance across diverse architectures such as ViT and ResNet on datasets including CIFAR and ImageNet-1K. Ultimately, this work offers a novel perspective on enhancing model performance by strategically capitalizing on the dynamics of the brief and crucial early stages of training. Code is available at https://anonymous.4open.science/r/code-A5F1/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。