用激进模态丢弃逆转多模态学习中的负向协同，提升模型性能。
Negative to Positive Co-learning with Aggressive Modality Dropout
- 通过激进模态丢弃策略，迫使模型关注互补信息。
- 在负向协同学习中最高提升20%准确率，显著改善性能。
- 适合需要鲁棒多模态训练的场景，如跨模态对齐任务。
本文提出一种通过激进模态丢弃来改进多模态协同学习的有效方法。研究发现，激进模态丢弃能将负向协同学习（NCL）转化为正向协同学习（PCL）。该方法可为单模态部署预训练多模态模型，并在负向协同学习中大幅提升性能，实验中最高实现20%的准确率提升。同时，与正向协同学习（PCL）对比表明，该方法虽在PCL中效果不如在NCL中显著，但仍能进一步改善协同学习表现。代码已开源。
原文摘要 · Abstract (English)
This paper aims to document an effective way to improve multimodal co-learning by using aggressive modality dropout. We find that by using aggressive modality dropout we are able to reverse negative co-learning (NCL) to positive co-learning (PCL). Aggressive modality dropout can be used to "prep" a multimodal model for unimodal deployment, and dramatically increases model performance during negative co-learning, where during some experiments we saw a 20% gain in accuracy. We also benchmark our modality dropout technique against PCL to show that our modality drop out technique improves co-learning during PCL, although it does not have as much as an substantial effect as it does during NCL. Github: https://github.com/nmagal/modality_drop_for_colearning
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。