Tangma通过可学习参数提升神经网络训练稳定性和效率。
Tangma: A Tanh-Guided Activation Function with Learnable Parameters
- 结合tanh平滑特性，引入两个可学习参数调节激活行为。
- 在MNIST上达99.09%准确率，CIFAR-10上达78.15%准确率。
- 相比Swish和GELU，训练更稳定且单轮耗时更低。
激活函数是深度神经网络中有效反向传播与表达能力的关键。本文提出Tangma，一种结合双曲正切平滑特性的新型激活函数，包含两个可学习参数：α用于移动曲线拐点以调节神经元激活，γ用于引入线性成分以保持微弱梯度并提升训练稳定性。在基于卷积与线性层构建的自定义网络上，对MNIST和CIFAR-10数据集进行了评估，并与ReLU、Swish和GELU对比。在MNIST上，Tangma达到99.09%最高验证准确率及最低验证损失，收敛速度更快且更稳定；在CIFAR-10上，其最高验证准确率达78.15%，优于所有基线模型，同时保持有竞争力的训练损失。Tangma还展现出更高的训练效率，平均每轮运行时间低于Swish和GELU。结果表明，Tangma在标准视觉任务中表现优异，支持可靠高效的训练。其可学习设计赋予对激活行为更强的控制力，可能有助于更大规模模型在图像识别或语言建模中的应用。
原文摘要 · Abstract (English)
Activation functions are key to effective backpropagation and expressiveness in deep neural networks. This work introduces Tangma, a new activation function that combines the smooth shape of the hyperbolic tangent with two learnable parameters: $α$, which shifts the curve's inflection point to adjust neuron activation, and $γ$, which adds linearity to preserve weak gradients and improve training stability. Tangma was evaluated on MNIST and CIFAR-10 using custom networks composed of convolutional and linear layers, and compared against ReLU, Swish, and GELU. On MNIST, Tangma achieved the highest validation accuracy of 99.09% and the lowest validation loss, demonstrating faster and more stable convergence than the baselines. On CIFAR-10, Tangma reached a top validation accuracy of 78.15%, outperforming all other activation functions while maintaining a competitive training loss. Tangma also showed improved training efficiency, with lower average epoch runtimes compared to Swish and GELU. These results suggest that Tangma performs well on standard vision tasks and enables reliable, efficient training. Its learnable design gives more control over activation behavior, which may benefit larger models in tasks such as image recognition or language modeling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。