用输入空间的区域数量揭示神经网络隐式偏置，解释为何某些训练方式泛化更好。
Understanding Nonlinear Implicit Bias via Region Counts in Input Space
- 以输入空间中相同预测标签的连通区域数衡量隐式偏置
- 区域数越少，决策边界越简单，泛化性能越好
- 大学习率、小批量等超参可减少区域数，提升模型表现
神经网络强大的泛化能力常归因于隐式偏置，但非线性场景下的定义与机制仍不明确。本文提出以输入空间中具有相同预测标签的连通区域数量来刻画隐式偏置。相比依赖参数的度量（如范数或归一化间隔），区域数由函数映射决定，且对重参数化不变，更适用于非线性、过参数化模型。实验发现，较小的区域数对应几何简单的决策边界，并与良好泛化性能高度相关。我们还观察到，较大的学习率和较小的批量大小等优秀超参设置能诱导更小的区域数。进一步建立了理论联系，解释了大学习率如何在神经网络中促成小区域数。
原文摘要 · Abstract (English)
One explanation for the strong generalization ability of neural networks is implicit bias. Yet, the definition and mechanism of implicit bias in non-linear contexts remains little understood. In this work, we propose to characterize implicit bias by the count of connected regions in the input space with the same predicted label. Compared with parameter-dependent metrics (e.g., norm or normalized margin), region count can be better adapted to nonlinear, overparameterized models, because it is determined by the function mapping and is invariant to reparametrization. Empirically, we found that small region counts align with geometrically simple decision boundaries and correlate well with good generalization performance. We also observe that good hyper-parameter choices such as larger learning rates and smaller batch sizes can induce small region counts. We further establish the theoretical connections and explain how larger learning rate can induce small region counts in neural networks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。