arXiv:2410.05016cs.LGstat.ML2024-10ICLR被引 14

不依赖数据增强的表格数据自监督学习方法,提升下游任务性能。

T-JEPA: Augmentation-Free Self-Supervised Learning for Tabular Data

  • 用特征子集预测另一子集的隐空间表示,实现无增强自监督学习。
  • 在分类与回归任务中显著优于直接使用原始数据的模型。
  • 可识别关键特征,适合缺乏标签的表格数据预训练场景。

自监督学习通常通过构建样本的多视图来预训练模型,以提升下游任务表现,但其依赖数据增强,而表格数据难以构造有效增强。本文提出一种新型无增强自监督学习方法T-JEPA,基于联合嵌入预测架构(JEPA),通过从一个特征子集的隐表示预测另一个子集的隐表示,从而在不使用数据增强的情况下学习丰富表征。将该方法用于预训练,并在此基础上训练多个深度分类器。实验表明,该方法在分类和回归任务中均显著优于直接在原始数据空间训练的模型。此外,T-JEPA使部分模型持续优于或匹配梯度提升决策树等传统方法。我们进一步分析发现,该方法能在无标签条件下有效识别下游任务相关特征。同时引入正则化标记(regularization tokens),作为训练结构化数据上JEPA模型的关键正则化手段。

原文摘要 · Abstract (English)

Self-supervision is often used for pre-training to foster performance on a downstream task by constructing meaningful representations of samples. Self-supervised learning (SSL) generally involves generating different views of the same sample and thus requires data augmentations that are challenging to construct for tabular data. This constitutes one of the main challenges of self-supervision for structured data. In the present work, we propose a novel augmentation-free SSL method for tabular data. Our approach, T-JEPA, relies on a Joint Embedding Predictive Architecture (JEPA) and is akin to mask reconstruction in the latent space. It involves predicting the latent representation of one subset of features from the latent representation of a different subset within the same sample, thereby learning rich representations without augmentations. We use our method as a pre-training technique and train several deep classifiers on the obtained representation. Our experimental results demonstrate a substantial improvement in both classification and regression tasks, outperforming models trained directly on samples in their original data space. Moreover, T-JEPA enables some methods to consistently outperform or match the performance of traditional methods likes Gradient Boosted Decision Trees. To understand why, we extensively characterize the obtained representations and show that T-JEPA effectively identifies relevant features for downstream tasks without access to the labels. Additionally, we introduce regularization tokens, a novel regularization method critical for training of JEPA-based models on structured data.

自监督学习表格数据特征识别无增强

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。