揭示神经网络中 Lipschitz 连续性的内在规律,提升模型鲁棒性与泛化能力。
Principles of Lipschitz continuity in neural networks
- 从训练动态角度分析 Lipschitz 连续性的演化过程
- 发现 Lipschitz 连续性影响输入频率信号的传播特性
- 为提升模型对扰动和分布外数据的鲁棒性提供理论依据
深度学习在多个领域取得显著成功,但对小输入扰动的鲁棒性及分布外数据的泛化能力仍面临挑战。这些问题凸显了理解神经网络鲁棒性与泛化本质原理的必要性。其中,Lipschitz 连续性在控制输出对输入扰动的最坏敏感度方面起关键作用。尽管其重要性广受认可,以往研究多集中于经验性正则化方法,缺乏对底层原理的深入探索。本文从内部视角(训练过程中 Lipschitz 连续性的演化)与外部视角(输入特征中频率信号传播的调控机制)两方面,系统研究 Lipschitz 连续性在机器学习范式下的基本原理,深化对其在神经网络中作用机制的理解。
原文摘要 · Abstract (English)
Deep learning has achieved remarkable success across a wide range of domains, significantly expanding the frontiers of what is achievable in artificial intelligence. Yet, despite these advances, critical challenges remain -- most notably, ensuring robustness to small input perturbations and generalization to out-of-distribution data. These critical challenges underscore the need to understand the underlying fundamental principles that govern robustness and generalization. Among the theoretical tools available, Lipschitz continuity plays a pivotal role in governing the fundamental properties of neural networks related to robustness and generalization. It quantifies the worst-case sensitivity of network's outputs to small input perturbations. While its importance is widely acknowledged, prior research has predominantly focused on empirical regularization approaches based on Lipschitz constraints, leaving the underlying principles less explored. This thesis seeks to advance a principled understanding of the principles of Lipschitz continuity in neural networks within the paradigm of machine learning, examined from two complementary perspectives: an internal perspective -- focusing on the temporal evolution of Lipschitz continuity in neural networks during training (i.e., training dynamics); and an external perspective -- investigating how Lipschitz continuity modulates the behavior of neural networks with respect to features in the input data, particularly its role in governing frequency signal propagation (i.e., modulation of frequency signal propagation).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。