arXiv:2606.15743cs.LG2026-06

解决多模态数据缺失问题,无需标注即可自动补全缺失模态特征。

Unsupervised Learning for Missing Modalities in Multimodal Learning

论文配图:Unsupervised Learning for Missing Modalities in Multimodal Learning
图 1 · 摘自论文原文
  • 通过无监督聚类和距离度量,公平处理不完整样本的跨模态结构。
  • 在超过50%模态缺失时仍保持F1-Micro超0.7,显著优于现有方法。
  • 框架轻量通用,可无缝集成到任意下游任务中,适合真实场景应用。

本文针对多模态学习中的缺失模态问题,提出一种名为UL4M4的无监督框架,可在不依赖具体任务的情况下,对缺失的特征嵌入进行补全。通过引入模态特异性归一化和新型部分模态距离度量,实现对不完整观测的公平聚类,保留跨模态结构并具备尺度不变性。聚类中心指导迭代贪婪补全过程,支持任意模态数量与任意缺失模式。补全模块轻量、使用冻结编码器,与下游任务解耦,可灵活接入任意融合/预测架构。大量实验表明,在多样且高度不完整的设置下,UL4M4表现稳健,据我们所知首次在极端缺失配置下实现一致的F1-Micro分数超过0.7(当超过50%模态槽缺失时),结果在不同聚类规模下稳定,显著超越当前最优基线。代码已公开于https://github.com/h-ismkhan/Multimodal-Learning-with-Missing-Modalities-via-Unsupervised-Learning。

原文摘要 · Abstract (English)

This paper addresses the missing-modality challenge in multi-modal learning by introducing Unsupervised Learning for Missing Modalities in Multi-Modal Learning (UL4M4), a flexible framework that imputes missing feature embeddings in a task-independent manner before supervised prediction. We propose modality-specific normalization and a novel partial-modality distance metric to enable fair clustering of incomplete observations, capturing cross-modal structures while preserving scale-invariance across varying dimensionalities and modality counts. Cluster centers from this unsupervised stage guide an iterative greedy imputation process for any missing modalities during training or inference, supporting arbitrary numbers of modalities and arbitrary missing patterns per sample. The imputation module is lightweight, uses frozen encoders, and decouples from the downstream task, allowing easy integration with any fusion/prediction architecture. Extensive experiments under diverse and highly incomplete regimes demonstrate UL4M4's robustness, achieving, to the best of our knowledge, the first consistent F1-Micro scores above 0.7 on challenging missing configurations even when more than 50\% of modality slots are missing. Results are also stable across cluster sizes and significantly outperform state-of-the-art baselines. Code is available here: https://github.com/h-ismkhan/Multimodal-Learning-with-Missing-Modalities-via-Unsupervised-Learning.

多模态学习缺失数据无监督学习特征补全

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。