用傅里叶与隐式类别特征处理，让深度学习在表格数据上追平甚至超越树模型。
Closing the gap on tabular data with Fourier and Implicit Categorical Features
- 通过统计方法识别离散化后与目标强相关的特征，提升类别特征利用效率。
- 引入可学习傅里叶变换，缓解神经网络对平滑解的偏好，更贴合数据本质。
- 在主流表格数据集上，深度模型性能接近或超过XGBoost，突破传统瓶颈。
尽管深度学习在多种数据类型上表现优异，但在表格数据上仍落后于树基方法，常被称为神经网络最后的‘未征服城堡’。我们假设树模型的优势在于其天然能够建模和利用由类别特征引发的非线性交互。相比之下，神经网络倾向于对特征进行均匀数值处理并输出平滑解，难以有效捕捉此类模式。为此，我们采用基于统计的特征处理技术，识别离散化后与目标强相关的特征。同时，通过引入可学习傅里叶变换，缓解深度模型对过度平滑解的偏好，使其更符合数据内在特性。实验表明，该预处理方法显著提升深度学习模型性能，使其在综合性表格数据基准上达到或超越XGBoost水平。
原文摘要 · Abstract (English)
While Deep Learning has demonstrated impressive results in applications on various data types, it continues to lag behind tree-based methods when applied to tabular data, often referred to as the last "unconquered castle" for neural networks. We hypothesize that a significant advantage of tree-based methods lies in their intrinsic capability to model and exploit non-linear interactions induced by features with categorical characteristics. In contrast, neural-based methods exhibit biases toward uniform numerical processing of features and smooth solutions, making it challenging for them to effectively leverage such patterns. We address this performance gap by using statistical-based feature processing techniques to identify features that are strongly correlated with the target once discretized. We further mitigate the bias of deep models for overly-smooth solutions, a bias that does not align with the inherent properties of the data, using Learned Fourier. We show that our proposed feature preprocessing significantly boosts the performance of deep learning models and enables them to achieve a performance that closely matches or surpasses XGBoost on a comprehensive tabular data benchmark.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。