用物理相变理论解析模型如何学不变特征,揭示正则化路径上的学习机制。
From Objectives to What Models Learn: A Landau Theory of Invariant Learning
- 将表示学习类比为磁性系统的多模态相变,推导出有效自由能模型。
- 发现低阶系数决定正则化相变类型,关键强度阈值可预测特征保留与淘汰。
- 适用于浅层与深层网络,且在矩阵扩展中保持预测力,适合研究模型泛化机制。
不变学习旨在获取跨环境保持预测性的表征,但其目标函数在正则化路径上的行为常不透明。本文将表示学习视为多模态磁化过程,从具体不变学习目标推导出类朗道的有效自由能,其低阶系数构成目标的特征签名,并引发不同的正则化相型。二次修正项移动相变边界并实现有限强度下的模式消除;四次修正项调节相变后振幅,通常在有限强度下留下残余加载;高阶结构控制非单调尾部、不稳定性和大正则化下的崩溃。在典型双线性模型中,该理论给出了闭式相边界和稳态加载,以及区分捷径模式与稳定模式的关键正则化强度,定义了选择性保留窗口。受控实验验证了预测的相边界、加载及正则化相型。在一维和两层ReLU网络中,相同特征签名仍可预测定性正则化路径行为,尽管深度导致尺度偏移。矩阵扩展将框架推广至耦合集体模式,提出谱相边界判据。该框架将低阶目标特征转化为对正则化相型乃至模型学习内容的预测。
原文摘要 · Abstract (English)
Invariant learning seeks representations that remain predictive across environments, yet the behavior of its objectives along the regularization path is often opaque. We address this objective-behavior gap by viewing representation learning as multimode magnetization and deriving, from concrete invariant-learning objectives, a Landau-type effective free energy whose low-order coefficients form objective signatures and induce distinct regularization phenotypes. Effective quadratic corrections move the phase boundary and enable finite-strength mode elimination; quartic corrections regulate post-onset amplitude and typically leave residual loading at finite strength; higher-order structure governs non-monotone tails, instability, and collapse at large regularization. In a canonical bilinear model, the theory yields closed-form phase boundaries and steady-state loadings, as well as distinct critical strengths for shortcut and stable modes that define a selective-retention window. Controlled experiments confirm the predicted phase boundaries, loadings, and regularization phenotypes. In one- and two-hidden-layer ReLU networks, the same signatures remain predictive of qualitative regularization-path behavior despite depth-dependent shifts in scale. A matrix extension generalizes the framework to coupled collective modes and yields a spectral phase-boundary criterion. Together, the framework turns low-order objective signatures into predictions of regularization phenotypes and, ultimately, of what models learn as regularization varies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。