arXiv:2501.14000cs.LGcs.AI2025-01

让每个神经元用不同激活函数,提升模型捕捉复杂模式的能力。

Local Control Networks (LCNs): Optimizing Flexibility in Neural Network Data Pattern Capture

  • 为每个神经元配置独立的B样条激活函数,实现灵活的局部控制。
  • 在图像任务中比MLP略优,比KAN快5%且性能更佳。
  • 适合需要高效高适应性的深度学习场景,尤其视觉与符号建模。

多层感知机(MLPs)通常在所有隐藏层节点使用相同的激活函数(如ReLU、Sigmoid、Tanh),这种一致性虽有效,但可能限制对复杂数据模式的捕捉能力。本文认为,全网统一激活函数并非最优,提出在每个节点采用不同激活函数以增强灵活性和自适应性。为此,我们引入局部控制网络(LCNs),利用B样条函数实现每个节点的差异化激活曲线。数学分析表明LCNs优于传统MLPs。实验显示,在计算机视觉任务中,LCNs相较MLP有微弱提升,较KAN性能高出约5%,同时计算效率更高;在基础机器学习任务中,比MLP提升1%,比KAN提升0.6%;在符号公式表示任务中表现与KAN相当,均优于MLP。结果表明,节点级激活多样性可显著提升性能与效率。

原文摘要 · Abstract (English)

The widespread use of Multi-layer perceptrons (MLPs) often relies on a fixed activation function (e.g., ReLU, Sigmoid, Tanh) for all nodes within the hidden layers. While effective in many scenarios, this uniformity may limit the networks ability to capture complex data patterns. We argue that employing the same activation function at every node is suboptimal and propose leveraging different activation functions at each node to increase flexibility and adaptability. To achieve this, we introduce Local Control Networks (LCNs), which leverage B-spline functions to enable distinct activation curves at each node. Our mathematical analysis demonstrates the properties and benefits of LCNs over conventional MLPs. In addition, we demonstrate that more complex architectures, such as Kolmogorov-Arnold Networks (KANs), are unnecessary in certain scenarios, and LCNs can be a more efficient alternative. Empirical experiments on various benchmarks and datasets validate our theoretical findings. In computer vision tasks, LCNs achieve marginal improvements over MLPs and outperform KANs by approximately 5\%, while also being more computationally efficient than KANs. In basic machine learning tasks, LCNs show a 1\% improvement over MLPs and a 0.6\% improvement over KANs. For symbolic formula representation tasks, LCNs perform on par with KANs, with both architectures outperforming MLPs. Our findings suggest that diverse activations at the node level can lead to improved performance and efficiency.

神经网络激活函数模型效率深度学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。