arXiv:2503.06277cs.CV2025-03CVPR被引 12

提出STiL框架,让图像和表格数据在少标签下更精准地学习任务相关特征。

STiL: Semi-supervised Tabular-Image Learning for Comprehensive Task-Relevant Information Exploration in Multimodal Classification

  • 通过解耦对比一致性模块,分离共享与模态特有信息。
  • 在自然与医疗图像数据集上超越现有最先进方法。
  • 适合医疗、金融等需融合图文数据的任务场景。

多模态图像-表格学习日益受到关注,但受限于标注数据不足。现有自监督学习(SSL)常因任务无关性导致下游任务特征不优。半监督学习(SemiSL)结合有标签与无标签数据,是潜在解决方案。然而,现有多模态SemiSL方法多聚焦于单模态或模态共享特征,忽视了对任务相关的模态特异性信息的利用,造成模态信息鸿沟。本文提出STiL框架,通过解耦对比一致性模块,学习跨模态不变表示的同时保留模态特异性信息。此外,提出基于分类器共识的伪标签生成策略及原型引导的标签平滑技术,提升无标签数据中任务相关特征的学习质量。在自然与医学图像数据集上的实验表明,STiL优于当前最先进的监督/自监督/半监督图像/多模态方法。代码已开源。

原文摘要 · Abstract (English)

Multimodal image-tabular learning is gaining attention, yet it faces challenges due to limited labeled data. While earlier work has applied self-supervised learning (SSL) to unlabeled data, its task-agnostic nature often results in learning suboptimal features for downstream tasks. Semi-supervised learning (SemiSL), which combines labeled and unlabeled data, offers a promising solution. However, existing multimodal SemiSL methods typically focus on unimodal or modality-shared features, ignoring valuable task-relevant modality-specific information, leading to a Modality Information Gap. In this paper, we propose STiL, a novel SemiSL tabular-image framework that addresses this gap by comprehensively exploring task-relevant information. STiL features a new disentangled contrastive consistency module to learn cross-modal invariant representations of shared information while retaining modality-specific information via disentanglement. We also propose a novel consensus-guided pseudo-labeling strategy to generate reliable pseudo-labels based on classifier consensus, along with a new prototype-guided label smoothing technique to refine pseudo-label quality with prototype embeddings, thereby enhancing task-relevant information learning in unlabeled data. Experiments on natural and medical image datasets show that STiL outperforms the state-of-the-art supervised/SSL/SemiSL image/multimodal approaches. Our code is available at https://github.com/siyi-wind/STiL.

多模态半监督图像表格特征解耦

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。