通过可学习的参数控制GELU门控硬度，实现平滑训练到ReLU模型的可控转换。
$λ$-GELU: Learning Gating Hardness for Controlled ReLU-ization in Deep Networks
- 提出λ-GELU模型，用参数λ调控门控函数陡度，实现平滑激活向ReLU的渐进转化。
- 实验显示不同层具有结构化硬度分布，且对初始化鲁棒，验证了可调控性。
- 支持训练后无损替换为ReLU，适合需兼容传统推理工具链的部署场景。
Gaussian Error Linear Unit（GELU）是广泛使用的平滑替代ReLU的激活函数，但许多部署、压缩和分析工具链更自然地适用于分段线性（ReLU型）网络。本文研究一种基于硬度参数λ的GELU形式：f(x;λ)=xΦ(λx)，其中Φ为标准正态累积分布函数，λ∈[1,∞)控制门控陡度。目标是将平滑的门控训练转化为通向可兼容ReLU模型的可控路径。学习λ存在挑战：直接更新导致动态不稳定与有效梯度衰减，因此引入约束重参数化与优化器感知更新策略。在涵盖MLP、CNN和Transformer的多种模型-数据集组合上，观察到分层的结构化硬度分布，并评估其在不同初始化下的鲁棒性。进一步研究了一种确定性ReLU化策略：逐步硬化学习到的门控，逼近一个原则性目标，实现训练后λ-GELU到ReLU的低扰动替换。总体而言，λ-GELU提供了一个最小且可解释的调控旋钮，连接平滑训练与以ReLU为中心的下游流程。
原文摘要 · Abstract (English)
Gaussian Error Linear Unit (GELU) is a widely used smooth alternative to Rectifier Linear Unit (ReLU), yet many deployment, compression, and analysis toolchains are most naturally expressed for piecewise-linear (ReLU-type) networks. We study a hardness-parameterized formulation of GELU, f(x;λ)=xΦ(λ x), where Φ is the Gaussian CDF and λ \in [1, infty) controls gate sharpness, with the goal of turning smooth gated training into a controlled path toward ReLU-compatible models. Learning λ is non-trivial: naive updates yield unstable dynamics and effective gradient attenuation, so we introduce a constrained reparameterization and an optimizer-aware update scheme. Empirically, across a diverse set of model--dataset pairs spanning MLPs, CNNs, and Transformers, we observe structured layerwise hardness profiles and assess their robustness under different initializations. We further study a deterministic ReLU-ization strategy in which the learned gates are progressively hardened toward a principled target, enabling a post-training substitution of λ-GELU by ReLU with reduced disruption. Overall, λ-GELU provides a minimal and interpretable knob to profile and control gating hardness, bridging smooth training with ReLU-centric downstream pipelines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。