arXiv:2608.24381cs.LG2026-08

自监督预训练在标签稀缺时提升有限,但能更好应对测试时缺失数据。

When Does Self-Supervised Pretraining Help Tabular Models? A Study of Label Scarcity and Missing Data

  • 用掩码恢复法做自监督预训练,提升模型对缺失数据的鲁棒性。
  • 在10%标签下平均AUC达0.8954,但与随机森林仍有差距(0.9015)。
  • 适合标签少且测试时有缺失数据的场景,尤其在随机缺失下表现更优。

自监督学习(SSL)在表格数据中前景广阔,但其在极端标签稀缺和测试时缺失数据下的效果仍不明确。本文在14个不同分类任务上评估了掩码-恢复的预训练目标,对比从零训练和经典基线。结果显示:尽管SSL平均优于从零训练,且在10%标签下达到0.8954 AUC(接近随机森林的0.9015),但增益在任务间差异大且不显著(p = 0.626)。相反,针对原生缺失数据的设计并未带来普遍收益,反而在高缺失率数据集上性能下降。然而,在测试时完全随机缺失(MCAR)和结构化缺失(MNAR)下,预训练模型平均分别提升0.0245和0.0418 AUC,且多数任务正向提升。进一步比较发现,该方法与VIME、SCARF、SubTab等主流预训练方式无显著差异(调整后p > 0.459),说明结果具普适性。

原文摘要 · Abstract (English)

Self-supervised learning (SSL) has emerged as a promising approach for tabular data, yet its efficacy under extreme label scarcity and test-time missingness remains under-explored. In this paper, we evaluate a mask-and-recover SSL pretraining objective against training from scratch and classical baselines across 14 diverse classification tasks. First, while SSL outperforms training from scratch on average and remains competitive with state-of-the-art tree ensembles (achieving ~0.8954 AUC vs. Random Forest's 0.9015 at 10% labels), the SSL-vs-scratch gains exhibit high inter-task variance and lack significance (p = 0.626 at both 5% and 10% labels). Second, contrary to the hypothesis that missing-value imputation objectives universally benefit datasets with native missingness, SSL yields the most reliable improvements on clean datasets, while frequently degrading performance on datasets with high inherent missingness. Third, despite this training variance, SSL-pretrained models achieve a higher average AUC than scratch-trained models under both test-time missingness completely at random (MCAR) injection (+0.0245 AUC, positive on 11 of 14 tasks) and structured missingness shifts (MNAR, +0.0418 AUC, positive on 8 of 14 tasks), though neither difference remains statistically significant after Holm-Bonferroni correction for multiple comparisons (adjusted p = 0.118 and p = 0.518, respectively). Fourth, comparing our mask-and-recover objective against three established tabular SSL baselines (VIME, SCARF, SubTab) under an identical encoder architecture, we find no significant difference from any of them (adjusted p = 0.459, p = 1.000, p = 1.000), indicating our findings reflect general properties of tabular SSL rather than idiosyncrasies of one particular pretext task.

自监督学习表格数据标签稀缺缺失数据

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。