提出高效提示方法，解决多模态数据缺失时模型持续学习的性能下降问题。
Efficient Prompting for Continual Adaptation to Missing Modalities
- 设计三类提示：模态特异、任务感知、任务特异，增强特征学习能力。
- 在三个公开数据集上优于现有方法，有效缓解灾难性遗忘和计算开销。
- 适合处理设备故障或隐私限制导致的数据缺失场景，提升模型鲁棒性。
真实应用中常因设备故障或隐私问题出现模态缺失。在下游数据集上微调预训练模型时，若存在缺失模态，性能会显著下降。现有方法通常将多种缺失情况合并训练恢复模块或对齐多模态特征，导致性能受限、计算成本高，并在连续学习环境中易引发灾难性遗忘。本文将动态缺失模态问题建模为持续学习任务，提出持续多模态缺失模态任务。为此，引入三类提示：模态特异性、任务感知、任务特异性提示，使模型能够学习模态内、模态间、任务内、任务间的特征。此外，提出对比式任务交互策略，显式学习不同模态间的提示关联。在三个公开数据集上进行了广泛实验，结果表明该方法持续优于当前最优方法。
原文摘要 · Abstract (English)
Missing modality issues are common in real-world applications, arising from factors such as equipment failures and privacy concerns. When fine-tuning pre-trained models on downstream datasets with missing modalities, performance can degrade significantly. Current methods often aggregate various missing cases to train recovery modules or align multimodal features, resulting in suboptimal performance, high computational costs, and the risk of catastrophic forgetting in continual environments where data arrives sequentially. In this paper, we formulate the dynamic missing modality problem as a continual learning task and introduce the continual multimodal missing modality task. To address this challenge efficiently, we introduce three types of prompts: modality-specific, task-aware, and task-specific prompts. These prompts enable the model to learn intra-modality, inter-modality, intra-task, and inter-task features. Furthermore, we propose a contrastive task interaction strategy to explicitly learn prompts correlating different modalities. We conduct extensive experiments on three public datasets, where our method consistently outperforms state-of-the-art approaches.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。