首次在均值场框架下分析瓶颈自编码器的非线性训练过程。
Beyond Linear and Overcomplete Regimes: A Mean-Field Analysis of Bottleneck Autoencoders
- 基于均值场理论推导出编码器与解码器的显式学习动态
- 有限宽度网络的损失轨迹与均值场轨迹高度一致，且收敛至最优解
- 为非线性瓶颈自编码器提供可解释的理论支撑，适合理论研究者
自编码器（AEs）通过将数据映射到低维潜在空间来学习表征，并最小化重构误差。尽管其在实践中表现良好，但理论理解仍主要局限于线性模型或无瓶颈的情形。本文研究了具有固定有限维瓶颈的非线性自编码器在均值场（MF）框架下的行为。我们推导出编码器和解码器的显式均值场学习动态，为非线性设置下的训练提供了可解析的刻画。结果表明，在有限时间范围内，使用随机梯度下降训练的有限宽度网络的实证风险以高概率紧随均值场风险轨迹。在最优状态下，有限宽度网络的风险收敛至均值场最优解，证明有限网络具备充分表达能力以逼近无限宽度解。
原文摘要 · Abstract (English)
Autoencoders (AEs) learn low-dimensional representations by mapping data into a latent space while minimizing reconstruction error. Despite their empirical success, theoretical understanding remains limited and largely restricted to linear models or settings without a bottleneck. In this work, we study nonlinear AEs with a fixed finite-dimensional bottleneck in the mean-field (MF) regime. We derive explicit MF learning dynamics for both encoder and decoder, providing a tractable characterization of training in the nonlinear setting. We show that, over finite time horizons, the empirical risk of finite-width networks trained with stochastic gradient descent closely tracks the MF risk trajectory with high probability. At optimality, we further establish that the finite-width risk converges to the MF optimum, demonstrating that finite networks are sufficiently expressive to approximate the infinite-width solution.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。