为工业级表格数据设计自监督预训练框架,提升分类精度与鲁棒性。
MaskTab: Scalable Masked Tabular Pretraining with Scaling Laws and Distillation for Industrial Classification

- 用可学习标记编码缺失值,区分结构缺失与随机丢失。
- 在工业数据集上比之前方法提升5.04% AUC和8.28% KS。
- 模型表示可高效压缩,适合低延迟、高可解释场景。
表格数据是金融、医疗等高风险决策系统的核心。但工业级表格数据通常维度高、缺失值多、标注稀缺。尽管基础模型已革新视觉与语言领域,表格学习仍依赖人工特征,缺乏通用自监督框架。我们提出MaskTab,一个专为工业级表格数据设计的统一预训练框架。该框架通过专用可学习标记编码缺失值,使模型能区分结构缺失与随机丢失。采用双路径架构联合优化掩码重建与任务特定监督,并引入MoE增强损失,动态路由特征至专业子网络。在工业级基准上,相比先前方法在严格缩放条件下提升5.04% AUC和8.28% KS。其表征还能有效蒸馏至轻量模型,在严苛延迟与可解释性约束下实现2.55% AUC和4.85% KS提升,同时增强对分布偏移的鲁棒性。研究证明,只要尊重表格数据的结构性特点,其同样可进行基础模型化处理。
原文摘要 · Abstract (English)
Tabular data forms the backbone of high-stakes decision systems in finance, healthcare, and beyond. Yet industrial tabular datasets are inherently difficult: high-dimensional, riddled with missing entries, and rarely labeled at scale. While foundation models have revolutionized vision and language, tabular learning still leans on handcrafted features and lacks a general self-supervised framework. We present MaskTab, a unified pre-training framework designed specifically for industrial-scale tabular data. MaskTab encodes missing values via dedicated learnable tokens, enabling the model to distinguish structural absence from random dropout. It jointly optimizes a hybrid supervised pre-training scheme--utilizing a twin-path architecture to reconcile masked reconstruction with task-specific supervision--and an MoE-augmented loss that adaptively routes features through specialized subnetworks. On industrial-scale benchmarks, it achieves +5.04% AUC and +8.28% KS over prior art under rigorous scaling. Moreover, its representations distill effectively into lightweight models, yielding +2.55% AUC and +4.85% KS under strict latency and interpretability constraints, while improving robustness to distribution shifts. Our work demonstrates that tabular data admits a foundation-model treatment--when its structural idiosyncrasies are respected.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。