arXiv:2501.07304cs.CVcs.LG2025-01

用图像和表格数据联合训练,让表格模型更准更省图。

Code and Pixels: Multi-Modal Contrastive Pre-training for Enhanced Tabular Data Analysis

  • 用对比学习+掩码建模融合表格与图像信息
  • 在两个数据集上比纯表格模型分别提升1.48%和2.38%
  • 适合缺图像但想提升表格分析能力的研究者

表格数据学习至关重要,因其提供结构化信息,对全面理解与决策不可或缺。本文提出多任务对比掩码表格建模(MT-CMTM),通过利用表格与对应图像间的相关性增强表格模型。该方法结合对比学习与掩码表格建模,采用一维残差卷积网络加注意力机制(1D-ResNet-CBAM)高效处理表格数据,无需依赖图像即可完成下游任务,避免昂贵的图像获取与处理。我们在DVM汽车数据集和新构建的HIPMP数据集(关联膜制备参数与图像)上评估,结果表明MT-CMTM相比从头训练的1D-ResNet-CBAM,在HIPMP上相对MSE降低1.48%,在DVM上绝对准确率提升2.38%。验证了其鲁棒性与在多模态学习中的潜力。

原文摘要 · Abstract (English)

Learning from tabular data is of paramount importance, as it complements the conventional analysis of image and video data by providing a rich source of structured information that is often critical for comprehensive understanding and decision-making processes. We present Multi-task Contrastive Masked Tabular Modeling (MT-CMTM), a novel method aiming to enhance tabular models by leveraging the correlation between tabular data and corresponding images. MT-CMTM employs a dual strategy combining contrastive learning with masked tabular modeling, optimizing the synergy between these data modalities. Central to our approach is a 1D Convolutional Neural Network with residual connections and an attention mechanism (1D-ResNet-CBAM), designed to efficiently process tabular data without relying on images. This enables MT-CMTM to handle purely tabular data for downstream tasks, eliminating the need for potentially costly image acquisition and processing. We evaluated MT-CMTM on the DVM car dataset, which is uniquely suited for this particular scenario, and the newly developed HIPMP dataset, which connects membrane fabrication parameters with image data. Our MT-CMTM model outperforms the proposed tabular 1D-ResNet-CBAM, which is trained from scratch, achieving a relative 1.48% improvement in relative MSE on HIPMP and a 2.38% increase in absolute accuracy on DVM. These results demonstrate MT-CMTM's robustness and its potential to advance the field of multi-modal learning.

表格分析多模态学习对比学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。