突破单调神经网络的激活函数限制,实现更优的逼近能力与训练稳定性。
Advancing Constrained Monotonic Neural Networks: Achieving Universal Approximation Beyond Bounded Activations
- 通过非负权约束与交替饱和激活,实现单调函数的通用逼近。
- 证明凸单调激活配合非正权约束同样可通用逼近,拓展理论边界。
- 新架构自动适配权重符号,无需重参数化,提升训练稳定性。
传统构建单调多层感知机(MLP)的方法依赖非负权约束和有界激活函数,带来显著优化挑战。本文推广了先前理论,证明仅需非负权约束与在两侧交替饱和的激活函数,即可使MLP成为单调函数的通用逼近器。同时揭示激活函数饱和侧与权重约束符号间的等价关系,进一步证明:使用凸单调激活函数并施加非正权约束的MLP,同样具备通用逼近能力,与非负权版本形成对比。该结果为前期工作的经验有效性提供理论支持,并暗示可能的架构简化。此外,我们提出一种替代方案,使网络能根据权重符号自适应调整激活,避免权重重参数化,改善初始化与训练稳定性。实验验证了理论结论的有效性,表明新方法优于传统单调架构。
原文摘要 · Abstract (English)
Conventional techniques for imposing monotonicity in MLPs by construction involve the use of non-negative weight constraints and bounded activation functions, which pose well-known optimization challenges. In this work, we generalize previous theoretical results, showing that MLPs with non-negative weight constraint and activations that saturate on alternating sides are universal approximators for monotonic functions. Additionally, we show an equivalence between the saturation side in the activations and the sign of the weight constraint. This connection allows us to prove that MLPs with convex monotone activations and non-positive constrained weights also qualify as universal approximators, in contrast to their non-negative constrained counterparts. Our results provide theoretical grounding to the empirical effectiveness observed in previous works while leading to possible architectural simplification. Moreover, to further alleviate the optimization difficulties, we propose an alternative formulation that allows the network to adjust its activations according to the sign of the weights. This eliminates the requirement for weight reparameterization, easing initialization and improving training stability. Experimental evaluation reinforces the validity of the theoretical results, showing that our novel approach compares favourably to traditional monotonic architectures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。