arXiv:2505.03923cs.LG2025-05ICML

用噪声扰动自动选最重要特征,无需调参或重训练

SAND: One-Shot Feature Selection with Additive Noise Distortion

  • 通过可学习增益+高斯噪声,让模型自动选出最相关特征
  • 在多个基准数据集上表现超越或持平现有方法
  • 适合需要快速部署、不想调参的机器学习应用

特征选择是数据驱动应用中的关键步骤,能降低输入维度以提升学习准确率、计算效率和可解释性。现有先进方法常需选择后重新训练并进行大量超参数调优,使用复杂。本文提出一种新颖的非侵入式特征选择层,给定目标特征数 $k$,可在神经网络训练过程中自动识别并选择 $k$ 个最相关信息特征。该方法极为简洁,无需修改损失函数、网络结构或后续重训练。其核心公式为:$ ilde{x}_i = a_i x_i + (1-a_i)z_i$,其中 $x_i$ 为输入特征,$ ilde{x}_i$ 为输出,$z_i$ 为高斯噪声,$a_i$ 为可学习增益,且满足 $ extstyle rac{ extstyle ext{sum}}{ extstyle i} a_i^2 = k$。该设计通过加权噪声扰动与增益归一化,诱导出自动聚类效应,使 $k$ 个 $a_i$ 值趋近于 1(保留有用特征),其余趋近于 0(剔除冗余特征)。尽管形式极简,该方法在标准基准数据集及一个新真实世界数据集上均达到或超越现有方法性能,且无需为 $k$ 进行超参数搜索或重新训练。在线性回归框架下的理论分析进一步验证了其有效性。本工作表明,简单性与高性能并非对立,为机器学习中的特征选择提供了一种强大而直接的工具。

原文摘要 · Abstract (English)

Feature selection is a critical step in data-driven applications, reducing input dimensionality to enhance learning accuracy, computational efficiency, and interpretability. Existing state-of-the-art methods often require post-selection retraining and extensive hyperparameter tuning, complicating their adoption. We introduce a novel, non-intrusive feature selection layer that, given a target feature count $k$, automatically identifies and selects the $k$ most informative features during neural network training. Our method is uniquely simple, requiring no alterations to the loss function, network architecture, or post-selection retraining. The layer is mathematically elegant and can be fully described by: \begin{align} \nonumber \tilde{x}_i = a_i x_i + (1-a_i)z_i \end{align} where $x_i$ is the input feature, $\tilde{x}_i$ the output, $z_i$ a Gaussian noise, and $a_i$ trainable gain such that $\sum_i{a_i^2}=k$. This formulation induces an automatic clustering effect, driving $k$ of the $a_i$ gains to $1$ (selecting informative features) and the rest to $0$ (discarding redundant ones) via weighted noise distortion and gain normalization. Despite its extreme simplicity, our method delivers state-of-the-art performance on standard benchmark datasets and a novel real-world dataset, outperforming or matching existing approaches without requiring hyperparameter search for $k$ or retraining. Theoretical analysis in the context of linear regression further validates its efficacy. Our work demonstrates that simplicity and performance are not mutually exclusive, offering a powerful yet straightforward tool for feature selection in machine learning.

特征选择神经网络自动筛选

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。