通过分层初始化加速深度监督网络训练，零初始化辅助头避免梯度干扰。
LION-DG: Layer-Informed Initialization with Deep Gradient Protocols for Accelerated Neural Network Training
- 为辅助分类器设计分层初始化，主干用He初始化，辅助头从零开始。
- 在CIFAR-10上提速8.3%，在CIFAR-100上提速11.3%，精度不降。
- 无需调参、无计算开销，适合部署于深监督架构的训练优化。
权重初始化对神经网络优化至关重要，但现有方法多忽略层级差异。本文研究带辅助分类器的深层监督架构，发现未训练的辅助头会因梯度干扰导致早期训练不稳定。提出LION-DG：对辅助分类器头进行零初始化，主干使用标准He初始化。理论证明此策略实现梯度唤醒：初始化时辅助梯度为零，随权重增长自然激活，提供无超参隐式预热。在DenseNet-DS与ResNet-DS结构上测试CIFAR-10和CIFAR-100数据集，结果表明：(1) DenseNet-DS在CIFAR-10上收敛速度提升8.3%且精度相当；(2) 结合LSUV与LION-DG达到最佳精度（CIFAR-10上81.92%）；(3) ResNet-DS在CIFAR-100上实现11.3%提速，侧接辅助设计有效。研究揭示了架构特异性权衡，为实践者提供清晰指导。该方法简单、零超参数、无额外计算开销。
原文摘要 · Abstract (English)
Weight initialization remains decisive for neural network optimization, yet existing methods are largely layer-agnostic. We study initialization for deeply-supervised architectures with auxiliary classifiers, where untrained auxiliary heads can destabilize early training through gradient interference. We propose LION-DG, a layer-informed initialization that zero-initializes auxiliary classifier heads while applying standard He-initialization to the backbone. We prove that this implements Gradient Awakening: auxiliary gradients are exactly zero at initialization, then phase in naturally as weights grow -- providing an implicit warmup without hyperparameters. Experiments on CIFAR-10 and CIFAR-100 with DenseNet-DS and ResNet-DS architectures demonstrate: (1) DenseNet-DS: +8.3% faster convergence on CIFAR-10 with comparable accuracy, (2) Hybrid approach: Combining LSUV with LION-DG achieves best accuracy (81.92% on CIFAR-10), (3) ResNet-DS: Positive speedup on CIFAR-100 (+11.3%) with side-tap auxiliary design. We identify architecture-specific trade-offs and provide clear guidelines for practitioners. LION-DG is simple, requires zero hyperparameters, and adds no computational overhead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。