分析28篇滤波特征选择论文,发现评估方式存在系统性偏差。
Bias in Filter Feature Selection Evaluation: A Meta-Analysis of Datasets, Baselines, and Experimental Design Choices

- 通过分析28篇高影响力论文,梳理评估设计中的关键变量
- 发现数据集、基线数量等三因素解释33%的性能差异
- 提出五项建议,提升未来特征选择研究的公平性
自1990年以来,各类应用中提出了大量特征选择方法。为验证新方法的有效性,需在至少一个数据集上与现有文献中的至少一个基线方法进行比较。近年来表格型深度学习和机器学习中的数据价值评估进展提示,新方法、算法和模型的评估可能受到有意或无意的偏见影响。我们假设特征选择(特别是滤波特征选择,FFS)也存在类似趋势。本研究旨在分析FFS相关研究,识别影响评估的因素,并提出更强的评估原则。通过对1994至2025年间28篇高影响力FFS论文的分析,研究反思了如何审查这些研究,总结出过程中的经验教训,并给出五项基于证据的未来评估建议。多变量线性回归分析显示,R²=0.33,意味着33%的新方法相对于选定基线的表现差异(胜率)可由数据集数量、基线数量和新方法数量解释。该结果属中等解释力,考虑到这是首次此类研究,具有积极意义。中等解释力的原因在于胜率还受领域成熟度、数据集与基线类型及回归模型简化程度等额外因素影响。
原文摘要 · Abstract (English)
Background: Since 1990 many feature selection methods have been proposed across heterogeneous applications. To validate the usefulness of a new method, it needs to be compared against at least one baseline method from the existing literature on a feature selection task using at least one dataset. Recent developments in tabular Deep Learning (DL) and data valuation in Machine Learning (ML) suggest that the evaluation of new methods, algorithms, and models may be consciously or unconsciously biased. We hypothesise that a similar trend exists in feature selection (FS), particularly in filter feature selection (FFS). The aim of this study is therefore to examine FFS studies to identify factors that influence the evaluation and that might consist entry point for biases in order to recommend stronger principles for FFS evaluation. Methods: We analyse a sample of 28 high profile FFS studies published between 1994 and 2025. The analysis provides reflections on how to examine FFS studies, highlights lessons learned throughout the process, and gives five evidence-based recommendations for future FFS evaluation. Results: Multivariate Linear Regression analysis achieved a score of $R^2=0.33$. It means that 33% of the variance in the performance of new methods against chosen baselines (win rate) is explained by the number of datasets (#Datasets), the number of baselines (#Baselines), and the number of new methods (#NewMethods). Discussion: $R^2=0.33$ is considered medium explanation; which is promising given that this is the first such study. The medium explanation result is due to the fact that win rate is influenced by additional factors such as the maturity of the feature selection domain, the type of datasets and baselines, and the simplicity of the regression model used to explain the relationship.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。