揭示深度集合与乔诺西池化实现通用性的最小嵌入维度
Embedding Dimension Lower Bounds for Universality of Deep Sets and Janossy Pooling
- 提出新方法推导对称神经网络嵌入维度下界
- 证明深度集合在 d>1 时最小维度仅差常数因子
- 首次给出 k>1 时高阶乔诺西池化的非平凡下界
在众多实际应用中，将对称性嵌入神经网络架构至关重要。以包含 n 个 d 维点的点云为例，网络需学习定义在 ℝ^d 中 n 个点构成集合上的函数，一种自然的不变性网络构建范式是乔诺西池化，它推广了流行的深度集合架构。本文研究该方法的通用性，特别是嵌入维度必须多大才能保证架构的通用性。通过一种新颖技术，我们证明了嵌入维度所需大小的新下界：对于深度集合，在所有 d > 1 的情况下，该下界精确到常数因子；对于 k-元乔诺西池化，当 k > 1 时，我们首次获得非平凡的下界。
原文摘要 · Abstract (English)
In many practical applications it is important to build symmetries into neural network architectures. Consider the important case of permutation symmetry on point clouds consisting of $n$ points in $d$ dimensions. In this case the network learns a function on a set of $n$ points in $\mathbb{R}^d$, and a natural paradigm for constructing invariant networks is Janossy pooling, which generalizes the popular Deep Sets architecture. We study the universality of this approach, in particular the important question of how large the embedding dimension must be to guarantee universality of this architecture. Specifically, using a novel technique, we prove new lower bounds on the required size of this embedding dimension. For Deep Sets, this gives the correct minimal dimension up to a constant factor for all $d > 1$. For $k$-ary Janossy pooling, we prove the first non-trivial lower bound on the required embedding dimension when $k > 1$.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。