TabPFN在数据有噪声时仍能保持高精度与稳定注意力,适合工业场景的表格预测。
Noise Immunity in In-Context Tabular Learning: An Empirical Robustness Analysis of TabPFN's Attention Mechanisms
- 通过注入噪声特征、增加样本量、提升标签错误率,测试模型鲁棒性。
- 即使标签错误率高达30%,ROC-AUC仍保持高位,注意力分布清晰且聚焦有效特征。
- 注意力热图与SHAP分析显示模型能自动过滤噪声,适合金融医疗等复杂数据场景。
表格式基础模型(TFMs)如TabPFN通过上下文学习(ICL)实现跨异构表格数据集的泛化,仅需单次前向传播即可完成预测,无需针对特定数据集更新参数。该范式在金融、医疗等工业领域极具吸引力,因重新训练专属模型成本高,且常见无关特征、相关特征组及标签噪声等问题。本文通过受控合成扰动,系统评估了TabPFN在二分类任务中的抗干扰能力:(i) 增加随机无关特征或非线性相关特征以扩展数据维度;(ii) 增加训练样本数量;(iii) 提升误标标签比例。除了预测性能外,还分析了注意力集中度与基于注意力的特征排序指标。结果表明,无论在何种数据缺陷下,TabPFN均表现出显著韧性:ROC-AUC维持高位,注意力结构清晰锐利,重要特征始终被优先关注。定性可视化(注意力热图、特征嵌入、SHAP图)进一步揭示,模型在多层中逐步聚焦有用特征,有效分离信号与噪声。综合结果表明,TabPFN具备在多种数据不完美情形下维持高性能与一致内部行为的能力。
原文摘要 · Abstract (English)
Tabular foundation models (TFMs) such as TabPFN (Tabular Prior-Data Fitted Network) are designed to generalize across heterogeneous tabular datasets through in-context learning (ICL). They perform prediction in a single forward pass conditioned on labeled examples without dataset-specific parameter updates. This paradigm is particularly attractive in industrial domains (e.g., finance and healthcare) where tabular prediction is pervasive. Retraining a bespoke model for each new table can be costly or infeasible in these settings, while data quality issues such as irrelevant predictors, correlated feature groups, and label noise are common. In this paper, we provide strong empirical evidence that TabPFN is highly robust under these sub-optimal conditions. We study TabPFN and its attention mechanisms for binary classification problems with controlled synthetic perturbations that vary: (i) dataset width by injecting random uncorrelated features and by introducing nonlinearly correlated features, (ii) dataset size by increasing the number of training rows, and (iii) label quality by increasing the fraction of mislabeled targets. Beyond predictive performance, we analyze internal signals including attention concentration and attention-based feature ranking metrics. Across these parametric tests, TabPFN is remarkably resilient: ROC-AUC remains high, attention stays structured and sharp, and informative features are highly ranked by attention-based metrics. Qualitative visualizations with attention heatmaps, feature-token embeddings, and SHAP plots further support a consistent pattern across layers in which TabPFN increasingly concentrates on useful features while separating their signals from noise. Together, these findings suggest that TabPFN is a robust TFM capable of maintaining both predictive performance and coherent internal behavior under various scenarios of data imperfections.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。