提出ZC-Swish激活函数，让无归一化网络在小批量下训练更稳定。
ZC-Swish: Stabilizing Deep BN-Free Networks for Edge and Micro-Batch Applications
- 设计零中心化的Swish变体，动态保持激活值均值接近零。
- 深度16的网络中，测试准确率达51.5%，远超标准Swish的随机水平。
- 适合内存受限或隐私保护场景，如医疗影像和联邦学习。
批归一化（BN）是深度学习的核心技术，但在小批量（如3D医学影像）和非独立同分布联邦学习中失效。移除BN虽可解决其限制，却常引发梯度消失、通道死亡等训练崩溃。本文发现，标准激活函数（如Swish、ReLU）因非零中心特性，会随网络加深导致激活均值持续偏移，加剧不稳定性。为此，提出零中心Swish（ZC-Swish），通过参数化设计动态将激活均值锚定于零。在深度8、16、32的无BN卷积网络上进行压力测试，结果显示：标准Swish在深度16以上性能退化至近随机水平，而ZC-Swish保持稳定的层间激活动态，在深度16时以种子42获得最高测试准确率51.5%。该方法为内存受限与隐私敏感场景提供了高效可靠的深层网络稳定方案。
原文摘要 · Abstract (English)
Batch Normalization (BN) is a cornerstone of deep learning, yet it fundamentally breaks down in micro-batch regimes (e.g., 3D medical imaging) and non-IID Federated Learning. Removing BN from deep architectures, however, often leads to catastrophic training failures such as vanishing gradients and dying channels. We identify that standard activation functions, like Swish and ReLU, exacerbate this instability in BN-free networks due to their non-zero-centered nature, which causes compounding activation mean-shifts as network depth increases. In this technical communication, we propose Zero-Centered Swish (ZC-Swish), a drop-in activation function parameterized to dynamically anchor activation means near zero. Through targeted stress-testing on BN-free convolutional networks at depths 8, 16, and 32, we demonstrate that while standard Swish collapses to near-random performance at depth 16 and beyond, ZC-Swish maintains stable layer-wise activation dynamics and achieves the highest test accuracy at depth 16 (51.5%) with seed 42. ZC-Swish thus provides a robust, parameter-efficient solution for stabilizing deep networks in memory-constrained and privacy-preserving applications where traditional normalization is unviable.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。