通过解析模型内部交互模式，揭示深度网络泛化能力的演化规律。
Technical Report: Quantifying and Analyzing the Generalization Power of a DNN
- 将DNN推理逻辑分解为少量与或交互模式，量化每种交互的泛化能力。
- 发现训练过程存在三阶段动态：先学简单泛化交互，后学复杂非泛化交互。
- 实验证明非泛化交互学习直接导致训练与测试损失差距。
本文提出一种分析深度神经网络（DNN）泛化能力的新视角，即在训练过程中直接解耦并分析模型编码的可泛化与不可泛化交互的动态变化。该研究基于可解释人工智能领域的最新理论成果，证明DNN的详细推理逻辑可严格重写为少量与-或交互模式。在此基础上，提出一种高效方法，用于量化每种交互的泛化能力，并发现交互泛化能力在训练中呈现明显的三阶段动态：初期主要去除噪声和非泛化交互，学习简单且可泛化的交互；第二、三阶段则逐步捕获越来越复杂的、难以泛化的交互。实验结果验证了非泛化交互的学习是训练与测试损失差距的直接原因。
原文摘要 · Abstract (English)
This paper proposes a new perspective for analyzing the generalization power of deep neural networks (DNNs), i.e., directly disentangling and analyzing the dynamics of generalizable and non-generalizable interaction encoded by a DNN through the training process. Specifically, this work builds upon the recent theoretical achievement in explainble AI, which proves that the detailed inference logic of DNNs can be can be strictly rewritten as a small number of AND-OR interaction patterns. Based on this, we propose an efficient method to quantify the generalization power of each interaction, and we discover a distinct three-phase dynamics of the generalization power of interactions during training. In particular, the early phase of training typically removes noisy and non-generalizable interactions and learns simple and generalizable ones. The second and the third phases tend to capture increasingly complex interactions that are harder to generalize. Experimental results verify that the learning of non-generalizable interactions is the the direct cause for the gap between the training and testing losses.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。