揭示了神经网络参数冗余的局部差异，提出两种新复杂度度量。
On Functional Dimension and Persistent Pseudodimension
- 定义局部功能维数与持久伪VC维，刻画参数空间复杂性
- 前者可基于小批量数据快速计算，后者可约束泛化误差
- 为理解双下降现象提供新视角，适合理论学习者
对于任意固定前馈ReLU神经网络结构，许多不同的参数设置可产生相同函数。这种冗余在参数空间中分布不均，尚不为人熟知。本文讨论两种适用于局部的复杂度度量：(1) 局部功能维数 [14, 18]，(2) 我们称为持久伪VC维的局部版VC维。前者可在有限样本批次上轻松计算；后者应能给出泛化差距的局部界，从而增进对双下降现象机制的理解。
原文摘要 · Abstract (English)
For any fixed feedforward ReLU neural network architecture, it is well-known that many different parameter settings can determine the same function. It is less well-known that the degree of this redundancy is inhomogeneous across parameter space. In this work, we discuss two locally applicable complexity measures for ReLU network classes and what we know about the relationship between them: (1) the local functional dimension [14, 18], and (2) a local version of VC dimension that we call persistent pseudodimension. The former is easy to compute on finite batches of points; the latter should give local bounds on the generalization gap, which would inform an understanding of the mechanics of the double descent phenomenon [7].
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。