用可学习激活取代归一化层，实现深度网络的完全自适应能力。
SaluNet: Enabling Total Plasticity in Normalization-Free Deep Networks
- 提出SALU激活函数，无需归一化即可稳定训练信号。
- 在批量大小为1时，模型仍保持93.44%（CIFAR-10）准确率。
- 适用于图像分类与Transformer，媲美甚至超越传统归一化结构。
归一化层如BatchNorm和LayerNorm长期被视为深度网络稳定训练的必要组件。本文表明，它们可被单一可学习激活机制完全替代。我们发现标准归一化会抑制参数的可塑性：当与归一化层结合时，可学习激活参数迅速失去适应能力。受此启发，提出SALU（饱和自适应线性单元）： SALU(x;a,b) = a x / √(1 + a b x²)，其中a>0, b>0。 该函数具有有界性、可学习性，提供内在信号稳定性，不依赖批统计或外部仿射参数。基于SALU，构建了SaluNet范式：用SALU替代归一化层，用SWALU和GALU替代标准激活。在ResNet-18上，SaluNet-C-18在无归一化情况下于CIFAR-10达到97.35%，在CIFAR-100达83.25%；在批量大小为1时，准确率仍分别为93.44%和76.23%，而传统归一化架构在此条件下失败。对于Transformer，SaluNet-T在CIFAR-10上从90.92%提升至91.01%，在CIFAR-100上从66.54%提升至68.10%。SaluNet-C-50在ImageNet-1K上以224×224分辨率达到78.67% Top-1准确率，288×288下达79.23%。结果表明，归一化层抑制了总可塑性——这一生物神经元固有的特性，使深度网络能有效学习。
原文摘要 · Abstract (English)
Normalization layers such as BatchNorm and LayerNorm have long been considered essential for stable training in deep networks. This work demonstrates that they can be fully replaced by a single learnable activation mechanism. We identify a plasticity suppression effect induced by standard normalization: learnable activation parameters rapidly lose adaptability when paired with normalization layers. Motivated by this observation, we introduce SALU (Saturated Adaptive Linear Unit), \[ \operatorname{SALU}(x;a,b) = \frac{a x}{\sqrt{1 + a b x^2}},\quad a>0,\; b>0 \] a bounded, learnable activation that provides intrinsic signal stabilization without relying on batch statistics or external affine parameters. Building on SALU, we propose SaluNet, a paradigm grounded in total plasticity: SALU replaces normalization layers, while SWALU and GALU replace standard activations. With ResNet-18, SaluNet-C-18 achieves 97.35\% on CIFAR-10 and 83.25\% on CIFAR-100 without normalization, maintaining 93.44\% and 76.23\% at batch size 1 where normalized architectures fail. For transformers, SaluNet-T improves over LayerNorm-GELU from 90.92\% to 91.01\% on CIFAR-10 and from 66.54\% to 68.10\% on CIFAR-100. SaluNet-C-50 reaches 78.67\% Top-1 on ImageNet-1K at $224\times224$, and $79.23\%$ at $288\times288$. These results suggest normalization layers suppress total plasticity, a property biological neurons inherently possess, enabling deep networks to learn effectively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。