揭示ReLU网络参数空间的连通性与奇点分布规律。
Topology and Geometry of the Learning Space of ReLU Networks: Connectivity and Singularities
- 基于有向无环图架构分析参数空间连通性机制。
- 发现瓶颈节点和平衡条件决定参数空间是否连通。
- 揭示奇点与网络拓扑结构的深层关联，适合研究训练动力学者。
理解前馈ReLU网络参数空间的性质对分析和引导训练动态至关重要。初始化后，在梯度流下训练会将参数空间限制到由ReLU激活函数的齐次性所引发的代数簇。本文针对一般有向无环图（DAG）架构的前馈ReLU网络，系统研究了参数空间的（不）连通性及其中奇点的存在性。通过扩展已有结果，我们全面刻画了连通性，强调了瓶颈节点以及特定子集相关的平衡条件的作用。研究明确表明，奇点与底层DAG及其诱导子网络的拓扑结构密切相关。我们讨论了奇点的可达性，并建立了其与可微剪枝之间的理论联系。通过简单的数值实验验证了理论结论。
原文摘要 · Abstract (English)
Understanding the properties of the parameter space in feed-forward ReLU networks is critical for effectively analyzing and guiding training dynamics. After initialization, training under gradient flow decisively restricts the parameter space to an algebraic variety that emerges from the homogeneous nature of the ReLU activation function. In this study, we examine two key challenges associated with feed-forward ReLU networks built on general directed acyclic graph (DAG) architectures: the (dis)connectedness of the parameter space and the existence of singularities within it. We extend previous results by providing a thorough characterization of connectedness, highlighting the roles of bottleneck nodes and balance conditions associated with specific subsets of the network. Our findings clearly demonstrate that singularities are intricately connected to the topology of the underlying DAG and its induced sub-networks. We discuss the reachability of these singularities and establish a principled connection with differentiable pruning. We validate our theory with simple numerical experiments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。