针对多模态缺失问题，提出高效参数提示方法
EPE-P: Evidence-based Parameter-efficient Prompting for Multimodal Learning with Missing Modalities
- 统一设计跨模态提示，减少冗余参数
- 引入证据损失函数，提升对缺失模态的不确定性处理能力
- 适合多模态模型在实际场景中应对数据缺失
现实中的多模态学习常面临模态缺失问题，无论训练还是测试阶段均可能发生。现有方法通常需为每种模态或缺失情况单独设计提示，导致结构复杂且参数量剧增。随着模态数量增加，参数冗余问题愈发严重。为此，我们提出基于证据的参数高效提示（EPE-P），一种适用于预训练多模态网络的新方法。该方法通过整合不同模态的提示信息，简化设计并降低参数复杂度。同时，提出基于证据的损失函数，更好地建模缺失模态带来的不确定性，增强模型决策能力。实验表明，EPE-P在效果与效率上均优于现有提示方法。代码已开源：https://github.com/Boris-Jobs/EPE-P_MLLMs-Robustness。
原文摘要 · Abstract (English)
Missing modalities are a common challenge in real-world multimodal learning scenarios, occurring during both training and testing. Existing methods for managing missing modalities often require the design of separate prompts for each modality or missing case, leading to complex designs and a substantial increase in the number of parameters to be learned. As the number of modalities grows, these methods become increasingly inefficient due to parameter redundancy. To address these issues, we propose Evidence-based Parameter-Efficient Prompting (EPE-P), a novel and parameter-efficient method for pretrained multimodal networks. Our approach introduces a streamlined design that integrates prompting information across different modalities, reducing complexity and mitigating redundant parameters. Furthermore, we propose an Evidence-based Loss function to better handle the uncertainty associated with missing modalities, improving the model's decision-making. Our experiments demonstrate that EPE-P outperforms existing prompting-based methods in terms of both effectiveness and efficiency. The code is released at https://github.com/Boris-Jobs/EPE-P_MLLMs-Robustness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。