用分层渐进冻结替代梯度估计算法，让深层二值神经网络训练更高效
Layerwise Progressive Freezing Enables STE-Free Training of Deep Binary Neural Networks
- 分层随机掩码逐步将浮点权重转为二值，仅反向传播未冻结部分
- 在ResNet-50上，CIFAR-10准确率比STE基准高18.0个百分点
- 适合追求低精度部署的深度模型研究者，尤其关注硬件友好型训练
我们研究渐进冻结作为从头训练二值网络的替代方案，以避免使用直通估计器（STE）。在受控训练条件下发现，全局渐进冻结虽适用于二值权重网络，但在全二值神经网络中因激活引起的梯度阻塞而失效。为此提出StoMPP（随机掩码部分渐进二值化），通过分层随机掩码逐步将可微分截断权重/激活替换为硬二值阶梯函数，同时仅对未冻结（截断）部分进行反向传播（无需直通估计器）。在相同最小训练配方下，StoMPP优于基于BinaryConnect的STE基线，性能提升随网络深度增加：对于ResNet-50二值神经网络，在CIFAR-10上提升18.0，CIFAR-100上提升13.5，ImageNet上提升3.8；ResNet-18分别提升3.1、4.7和1.3。对于二值权重网络，使用ResNet-50在CIFAR-10上达到91.2%准确率，CIFAR-100上达69.5%。我们分析了渐进冻结下的训练动态，揭示非单调收敛特性及在二值化约束下更好的深度扩展性。
原文摘要 · Abstract (English)
We investigate progressive freezing as an alternative to straight-through estimators (STE) for training binary networks from scratch. Under controlled training conditions, we find that while global progressive freezing works for binary-weight networks, it fails for full binary neural networks due to activation-induced gradient blockades. We introduce StoMPP (Stochastic Masked Partial Progressive Binarization), which uses layerwise stochastic masking to progressively replace differentiable clipped weights/activations with hard binary step functions, while only backpropagating through the unfrozen (clipped) subset (i.e., no straight-through estimator). Under a matched minimal training recipe, StoMPP improves accuracy over a BinaryConnect-style STE baseline, with gains that increase with depth (e.g., for ResNet-50 BNN: +18.0 on CIFAR-10, +13.5 on CIFAR-100, and +3.8 on ImageNet; for ResNet-18: +3.1, +4.7, and +1.3). For binary-weight networks, StoMPP achieves 91.2\% accuracy on CIFAR-10 and 69.5\% on CIFAR-100 with ResNet-50. We analyze training dynamics under progressive freezing, revealing non-monotonic convergence and improved depth scaling under binarization constraints.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。