揭示深度网络与ICA在高维非高斯数据下特征学习的机制差异
Feature learning from non-Gaussian inputs: the case of Independent Component Analysis in high dimensions
- 用FastICA和SGD对比研究高维非高斯输入下的特征提取机制
- 证明FastICA需至少n≈d⁴样本才能恢复一个非高斯方向
- 指出图像强非高斯性可弥补FastICA样本效率低的问题
深度神经网络能从复杂非高斯输入中学习结构化特征，但其机制尚不明确。观察发现，深层卷积网络第一层滤波器与独立成分分析（ICA）所学滤波器相似，而ICA旨在寻找输入中最非高斯的投影。这提示ICA可作为研究特征学习的简洁且有原则的模型。本文利用此联系，研究主流ICA算法FastICA与训练深度网络的随机梯度下降（SGD）在特征学习中的交互。我们严格证明，在简单合成数据模型下，FastICA需至少n≳d⁴样本才能从d维输入中恢复单一非高斯方向。我们发现，原始在线SGD性能优于FastICA，且通过平滑损失函数可实现最优样本复杂度n≳d²，但依赖数据分布。最后，我们在ImageNet上发现FastICA存在搜索相，并讨论图像强非高斯性如何补偿其样本效率不足。
原文摘要 · Abstract (English)
Deep neural networks learn structured features from complex, non-Gaussian inputs, but the mechanisms behind this process remain poorly understood. Our work is motivated by the observation that the first-layer filters learnt by deep convolutional neural networks from natural images resemble those learnt by independent component analysis (ICA), a simple unsupervised method that seeks the most non-Gaussian projections of its inputs. This similarity suggests that ICA provides a simple, yet principled model for studying feature learning. Here, we leverage this connection to investigate the interplay between data structure and optimisation in feature learning for the most popular ICA algorithm, FastICA, and stochastic gradient descent (SGD), which is used to train deep networks. We rigorously establish that FastICA requires at least $n\gtrsim d^4$ samples to recover a single non-Gaussian direction from $d$-dimensional inputs on a simple synthetic data model. We show that vanilla online SGD outperforms FastICA, and prove that the optimal sample complexity $n \gtrsim d^2$ can be reached by smoothing the loss, albeit in a data-dependent way. We finally demonstrate the existence of a search phase for FastICA on ImageNet, and discuss how the strong non-Gaussianity of said images compensates for the poor sample complexity of FastICA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。