揭示层归一化与动态激活函数的数学联系，提出更精确的替代方案。
On the Mathematical Relationship Between Layer Normalization and Dynamic Activation Functions
- 从RMSNorm推导出动态激活函数DyT，需解耦与近似
- 提出精确版DyISRU，对异常值归一化更准确
- 适合研究归一化机制或模型优化的工程师
层归一化（LN）是现代神经网络的核心组件。尽管已有多种替代方法被提出，但尚未有成功取代LN的方案。最近的研究提出一种名为动态双曲正切（DyT）的动态激活函数，虽具实践吸引力，但缺乏理论基础。本文揭示了LN与动态激活函数间的数学关系：我们从RMSNorm变体出发，推导出DyT，发现需在导数空间进行解耦并引入近似。若直接在函数空间应用相同解耦过程，可省略近似，得到RMSNorm的精确逐元素对应形式——动态反平方根单元（DyISRU）。数值实验表明，相较于DyT，DyISRU对异常值的归一化效果更准确。
原文摘要 · Abstract (English)
Layer normalization (LN) is an essential component of modern neural networks. While many alternative techniques have been proposed, none of them have succeeded in replacing LN so far. The latest suggestion in this line of research is a dynamic activation function called Dynamic Tanh (DyT). Although it is empirically well-motivated and appealing from a practical point of view, it lacks a theoretical foundation. In this work, we shed light on the mathematical relationship between LN and dynamic activation functions. In particular, we derive DyT from the LN variant RMSNorm, and show that a well-defined decoupling in derivative space as well as an approximation are needed to do so. By applying the same decoupling procedure directly in function space, we are able to omit the approximation and obtain the exact element-wise counterpart of RMSNorm, which we call Dynamic Inverse Square Root Unit (DyISRU). We demonstrate numerically that DyISRU reproduces the normalization effect on outliers more accurately than DyT does.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。