比较三种实现不变性的方法，发现正则化更优但需注意临界点问题。
Understanding Learning Invariance in Deep Linear Networks
- 对比数据增强、正则化和硬编码三种不变性实现方式
- 正则化路径连续且收敛到硬编码解，但引入额外鞍点
- 硬编码与数据增强的临界点相同，仅含鞍点和全局最优
等变和不变机器学习模型利用数据中的对称性和结构模式以提高样本效率。尽管实证研究表明，如正则化和数据增强这类数据驱动方法可与显式不变模型相媲美，但理论理解仍不充分。本文针对深度线性网络中的均方误差回归，系统比较了三种实现不变性的方法：数据增强、正则化与硬编码。深度线性网络参数化低秩线性映射，可被硬编码为对特定群作用不变。我们证明，硬编码与数据增强优化问题的临界点完全相同，仅包含鞍点和全局最优解；而正则化引入额外临界点，但除全局最优外均为鞍点。此外，我们证明正则化路径连续，并收敛至硬编码解。
原文摘要 · Abstract (English)
Equivariant and invariant machine learning models exploit symmetries and structural patterns in data to improve sample efficiency. While empirical studies suggest that data-driven methods such as regularization and data augmentation can perform comparably to explicitly invariant models, theoretical insights remain scarce. In this paper, we provide a theoretical comparison of three approaches for achieving invariance: data augmentation, regularization, and hard-wiring. We focus on mean squared error regression with deep linear networks, which parametrize rank-bounded linear maps and can be hard-wired to be invariant to specific group actions. We show that the critical points of the optimization problems for hard-wiring and data augmentation are identical, consisting solely of saddles and the global optimum. By contrast, regularization introduces additional critical points, though they remain saddles except for the global optimum. Moreover, we demonstrate that the regularization path is continuous and converges to the hard-wired solution.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。