解决表格图像融合中的梯度冲突问题，提升多模态学习效果。
Harmonized Tabular-Image Fusion via Gradient-Aligned Alternating Learning
- 交替训练单模态与共享分类器，解耦模态梯度。
- 通过不确定性引导的梯度手术，选择性对齐跨模态梯度。
- 在多个数据集上超越当前最佳方法，适合多模态融合研究者。
多模态表格-图像融合是多个领域日益关注的任务。然而，现有方法可能因模态间梯度冲突而阻碍单模态学习优化。本文提出一种新型梯度对齐交替学习（GAAL）范式，通过对齐模态梯度来解决此问题。具体而言，GAAL采用交替单模态学习与共享分类器机制，解耦多模态梯度并促进交互；此外，设计基于不确定性的跨模态梯度手术，选择性对齐跨模态梯度，引导共享参数同时惠及各模态。实验表明，该方法在多个主流数据集上优于多种先进基线及测试时表格缺失场景下的基线，显著提升融合性能。代码已开源。
原文摘要 · Abstract (English)
Multimodal tabular-image fusion is an emerging task that has received increasing attention in various domains. However, existing methods may be hindered by gradient conflicts between modalities, misleading the optimization of the unimodal learner. In this paper, we propose a novel Gradient-Aligned Alternating Learning (GAAL) paradigm to address this issue by aligning modality gradients. Specifically, GAAL adopts an alternating unimodal learning and shared classifier to decouple the multimodal gradient and facilitate interaction. Furthermore, we design uncertainty-based cross-modal gradient surgery to selectively align cross-modal gradients, thereby steering the shared parameters to benefit all modalities. As a result, GAAL can provide effective unimodal assistance and help boost the overall fusion performance. Empirical experiments on widely used datasets reveal the superiority of our method through comparison with various state-of-the-art (SoTA) tabular-image fusion baselines and test-time tabular missing baselines. The source code is available at https://github.com/njustkmg/ICME26-GAAL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。