低层参数决定神经网络混淆样本，揭示模型泛化差异根源
Randomness of Low-Layer Parameters Determines Confusing Samples in Terms of Interaction Representations of a DNN
- 通过分析网络交互复杂度，发现低层参数主导混淆样本形成
- 不同低层参数导致完全不同的混淆样本集，即使整体性能相似
- 为彩票抽奖假说提供新解释，适合研究模型泛化机制的读者
本文发现，深度神经网络（DNN）所编码的交互复杂度可解释其泛化能力。我们进一步发现，混淆样本（即非泛化性交互所表示的样本）由网络低层参数决定。相比之下，高层参数与网络架构对混淆样本组成的影响较小。即使两个DNN性能相近，只要低层参数不同，其混淆样本集合也完全不同。这一发现拓展了彩票抽奖假说的理解，有效解释了不同DNN间表征能力的差异。
原文摘要 · Abstract (English)
In this paper, we find that the complexity of interactions encoded by a deep neural network (DNN) can explain its generalization power. We also discover that the confusing samples of a DNN, which are represented by non-generalizable interactions, are determined by its low-layer parameters. In comparison, other factors, such as high-layer parameters and network architecture, have much less impact on the composition of confusing samples. Two DNNs with different low-layer parameters usually have fully different sets of confusing samples, even though they have similar performance. This finding extends the understanding of the lottery ticket hypothesis, and well explains distinctive representation power of different DNNs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。