用傅里叶变换预处理,让神经网络在数据少时也能更好捕捉特征依赖关系。
Fourier Preconditioning for Neural Feature Learning
- 引入傅里叶变换作为无训练预处理器,优化特征学习的基底选择。
- 在八组多变量数据上实现最高50%的归一化均方误差降低。
- 提出谱熵和累积依赖能量指标,可提前预测预处理效果。
受互信息启发的特征学习方法能生成保留非线性依赖结构的低维嵌入,但直接估计互信息在小样本情况下易受概率分布估计噪声影响。基于二阶统计量的H-Score提供了一种实用的训练代理指标。我们证明,在无限制函数空间下H-Score对可逆变换不变,但在受限近似类中对输入基旋转敏感。因此,我们研究了H-Score网络的酉预处理,发现合适的基旋转可通过集中预测依赖于少数主导模式来减少有限宽度截断误差。我们识别出快速傅里叶变换(FFT)是近似平稳过程的有效、数据无关且低成本的预处理器,其谱结构促使交叉协方差奇异值谱集中。我们引入无需训练的谱熵与累积依赖能量指标,以量化基底适用性并预测下游推理收益。在八个多变量数据集上的实验表明,FFT预处理在资源受限场景下尤为有效,最多实现50%的归一化均方误差(NMSE)降低,且所提指标与实际性能提升高度相关,能准确识别谱预处理有害的情形。
原文摘要 · Abstract (English)
Mutual information (MI)-inspired feature learning techniques are capable of generating low-dimensional embeddings that retain nonlinear dependence structures, but direct estimations of MI suffer from noisy probability distribution estimates in the low-data regime. The H-Score objective, computed from second-order statistics, provides a practical proxy metric for training feature extraction networks. We prove that H-Score is invariant to invertible transformations in the unrestricted functional setting, but becomes sensitive to input basis rotations under constrained approximation classes. Consequently, we study unitary preconditioning for H-Score networks and show that selecting an appropriate basis rotation reduces finite-width truncation error by concentrating predictive dependence into fewer dominant modes. We identify the fast Fourier transform (FFT) as an effective data-independent, low-cost preconditioner for approximately stationary processes, where spectral structure induces concentration of the cross-covariance singular value spectrum. We introduce training-free metrics based on spectral entropy and cumulative dependence energy to quantify basis suitability and predict downstream inference gains prior to network training. Experiments across eight multivariate datasets demonstrate that FFT preconditioning is particularly useful in resource-constrained regimes, achieving up to 50% normalized mean squared error (NMSE) reduction, while the proposed metrics correlate with observed performance gains and correctly identify cases where spectral preconditioning is detrimental.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。