用序理论构建可解释的神经网络,支持高效池化与模块化设计。
Order Theory in the Context of Machine Learning
- 基于偏序集构造神经网络,其几何结构对应有序多面体。
- 四点偏序集生成2×2卷积滤波器,可直接嵌入任意网络架构。
- 无需额外训练参数,实现比传统池化更精确的梯度更新。
本文研究了整数权重但实数偏置的神经网络(IVNN)与热带有理函数之间的等价性,后者可映射到多面体。每个含n个元素的偏序集(poset)对应一个位于单位立方体[0,1]^n内的有序多面体,其坐标满足偏序不等式。本文关注那些关联多面体为有序多面体的神经网络,并发现四个点的偏序集可诱导出2×2卷积滤波器,这些滤波器可应用于任意神经网络,不限于IVNN。类似Maxout,这类偏序池化滤波器在反向传播中以更高精度更新权重,且无需训练额外参数。实验验证了相关结论。此外,本文定义了偏序集上的运算符代数结构,用于研究偏序神经网络与热带多项式的组合关系,引入多面体的闵可夫斯基和与凸包包络作为关键操作,分析其对对应牛顿多面体的影响。
原文摘要 · Abstract (English)
The paper ``Tropical Geometry of Deep Neural Networks'' by L. Zhang et al. introduces an equivalence between integer-valued neural networks (IVNN) with $\text{ReLU}_{t}$ and tropical rational functions, which come with a map to polytopes. Here, IVNN refers to a network with integer weights but real biases, and $\text{ReLU}_{t}$ is defined as $\text{ReLU}_{t}(x)=\max(x,t)$ for $t\in\mathbb{R}\cup\{-\infty\}$. For every poset with $n$ points, there exists a corresponding order polytope, i.e., a convex polytope in the unit cube $[0,1]^n$ whose coordinates obey the inequalities of the poset. We study neural networks whose associated polytope is an order polytope. We then explain how posets with four points induce neural networks that can be interpreted as $2\times 2$ convolutional filters. These poset filters can be added to any neural network, not only IVNN. Similarly to maxout, poset pooling filters update the weights of the neural network during backpropagation with more precision than average pooling, max pooling, or mixed pooling, without the need to train extra parameters. We report experiments that support our statements. We also define the structure of algebra over the operad of posets on poset neural networks and tropical polynomials. This formalism allows us to study the composition of poset neural network arquitectures and the effect on their corresponding Newton polytopes, via the introduction of the generalization of two operations on polytopes: the Minkowski sum and the convex envelope.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。