用非参数统计方法实现深度网络特征选择,理论保证强。
A Nonparametric Statistics Approach to Feature Selection in Deep Neural Networks with Theoretical Guarantees
- 基于二阶Stein公式,不依赖梯度下降选特征
- 样本量n=Ω(p²)时可保证选对全部相关特征
- 高维下n=Ω(s log p)仍有效,适合复杂交互数据
本文研究深度神经网络中特征选择问题,设定条件为输出期望仅依赖于一组稀疏相关特征($oldsymbol{x}_{oldcal{S}_0}$),且关系函数$G$未知但满足弱光滑性。提出将神经网络重构成索引模型,利用二阶Stein公式估计相关特征集$oldsymbol{S}_0$,无需梯度下降,可在样本量满足$n = Ω(p^2)$时保证选择一致性。针对高维情形,引入筛选-选择机制,在样本量满足$n = Ω(s \ log p)$时实现非线性选择一致性,其中$s$为真实相关特征数。进一步在选中特征上重新训练神经网络进行预测,并在较弱稀疏假设下建立性能保证。大量模拟与真实数据分析表明,该方法在复杂特征交互场景下仍表现优异。
原文摘要 · Abstract (English)
This paper tackles the problem of feature selection in a highly challenging setting: $\mathbb{E}(y | \boldsymbol{x}) = G(\boldsymbol{x}_{\mathcal{S}_0})$, where $\mathcal{S}_0$ is the set of relevant features and $G$ is an unknown, potentially nonlinear function subject to mild smoothness conditions. Our approach begins with feature selection in deep neural networks, then generalizes the results to H{ö}lder smooth functions by exploiting the strong approximation capabilities of neural networks. Unlike conventional optimization-based deep learning methods, we reformulate neural networks as index models and estimate $\mathcal{S}_0$ using the second-order Stein's formula. This gradient-descent-free strategy guarantees feature selection consistency with a sample size requirement of $n = Ω(p^2)$, where $p$ is the feature dimension. To handle high-dimensional scenarios, we further introduce a screening-and-selection mechanism that achieves nonlinear selection consistency when $n = Ω(s \log p)$, with $s$ representing the sparsity level. Additionally, we refit a neural network on the selected features for prediction and establish performance guarantees under a relaxed sparsity assumption. Extensive simulations and real-data analyses demonstrate the strong performance of our method even in the presence of complex feature interactions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。