arXiv:2409.13342stat.MLcs.LG2024-09被引 6

低性能模型也能有效分析特征重要性,关键在数据规模充足。

Validity of Feature Importance in Low-Performing Machine Learning for Tabular Biomedical Data

  • 通过数据裁剪与特征裁剪对比,检验模型性能下降时特征排序稳定性。
  • 真实数据中特征裁剪比数据裁剪更稳定,尤其在控制特征相关性后。
  • 即使模型准确率低,只要数据量足够,特征重要性仍可信,适合临床辅助分析。

在表格式生物医学数据分析中,通常认为模型高精度是讨论特征重要性的前提。本文挑战这一共识,表明低性能模型也可用于特征重要性分析。通过三个合成数据集和六个真实生物医学数据集,我们比较完整数据与样本量减少(数据裁剪)或特征数减少(特征裁剪)下的特征排名变化。在合成数据中,特征裁剪不改变特征排名,而数据裁剪随性能下降导致更大偏差;真实数据中,特征裁剪的稳定性优于或等于数据裁剪,部分数据集则相反。当控制特征交互(移除相关性)后,特征裁剪始终更稳定。通过分析特征重要性分布及模型区分能力的理论概率,发现特征裁剪下模型仍能有效区分特征重要性,而数据裁剪则不能。结论:只要数据量充足,即使模型性能较低,特征重要性依然有效。这为在分类器表现不佳时结合统计分析进行特征相对比较提供了可能。

原文摘要 · Abstract (English)

In tabular biomedical data analysis, tuning models to high accuracy is considered a prerequisite for discussing feature importance, as medical practitioners expect the validity of feature importance to correlate with performance. In this work, we challenge the prevailing belief, showing that low-performing models may also be used for feature importance. We propose experiments to observe changes in feature rank as performance degrades sequentially. Using three synthetic datasets and six real biomedical datasets, we compare the rank of features from full datasets to those with reduced sample sizes (data cutting) or fewer features (feature cutting). In synthetic datasets, feature cutting does not change feature rank, while data cutting shows higher discrepancies with lower performance. In real datasets, feature cutting shows similar or smaller changes than data cutting, though some datasets exhibit the opposite. When feature interactions are controlled by removing correlations, feature cutting consistently shows better stability. By analyzing the distribution of feature importance values and theoretically examining the probability that the model cannot distinguish feature importance between features, we reveal that models can still distinguish feature importance despite performance degradation through feature cutting, but not through data cutting. We conclude that the validity of feature importance can be maintained even at low performance levels if the data size is adequate, which is a significant factor contributing to suboptimal performance in tabular medical data analysis. This paper demonstrates the potential for utilizing feature importance analysis alongside statistical analysis to compare features relatively, even when classifier performance is not satisfactory.

特征重要性低性能模型生物医学数据稳定性分析

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。