解决图文模态缺失下的细粒度对齐问题,提升分类准确率
Disentangled Fine-Grained Prototype Learning for Incomplete Image-Tabular Classification

- 设计共享-特定原型模型,分离跨模态与单模态特征
- 通过原型级对齐实现细粒度分布与语义一致性,精度提升5.2%以上
- 适合处理医疗、推荐等存在数据缺失的多模态场景
缺失模态问题是图像-表格多模态学习中的重大挑战,广泛存在于产品理解、推荐系统和医学诊断等应用中。当两种模态高度异构时(如图像与表格属性在语义粒度和数据分布上差异显著),现有方法仅基于全局特征平均进行解耦与对齐,仅捕捉粗粒度跨模态一致性,忽视了细粒度语义与分布错位,限制了缺失模态下互补信息的利用。为此,我们提出DFPL框架:首先,共享-特定原型建模(SSPM)提取紧凑且多样化的共享与模态特定原型,并在原型层面进行解耦以抑制模态内冗余相关性;其次,提出原型引导的细粒度对齐(PFA)模块,在统一原型空间中联合强制原型级分布匹配与原型到类的语义对齐,保留细粒度分布与语义一致性;此外,引入类别感知多尺度聚合(CMA)模块,自适应融合全局与原型层级的共享语义与模态特异性特征,实现鲁棒预测。在三个多样化图像-表格基准上的大量实验表明,本方法在多种缺失模态设置下均优于先前方法。代码将公开。
原文摘要 · Abstract (English)
The missing-modality problem poses a significant challenge in image-tabular multimodal learning across a wide range of multimedia applications, including product understanding, recommendation systems, and medical diagnosis. This challenge is particularly pronounced when the two modalities are highly heterogeneous, as images and tabular attributes differ substantially in their semantic granularity and data distributions. Existing methods learn modality-invariant representations through disentanglement and alignment over global token-averaged features, capturing only coarse cross-modal consistency and overlooking fine-grained semantic and distributional misalignment, which hampers the exploitation of complementary cues under missing modalities. To address this, we propose DFPL, a novel framework for fine-grained prototype learning. Specifically, Shared-Specific Prototype Modeling (SSPM) extracts compact and diverse shared and modality-specific prototypes, and further performs prototype-level disentanglement to suppress redundant intra-modality correlations. Additionally, we propose a Prototype-guided Fine-grained Alignment (PFA) module that jointly enforces prototype-level distribution matching and prototype-to-class semantic alignment within a unified prototype space, thereby preserving both fine-grained distributional and semantic consistency across modalities. We further introduce a Class-aware Multi-scale Aggregation (CMA) module to adaptively aggregate shared semantics and modality-specific characteristics from global and prototype levels for robust predictions. Extensive experiments on three diverse image-tabular benchmarks demonstrate the superiority of our method compared to the previous approaches under various missing-modality settings. Code will be made publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。