让视觉模型在缺失模态时仍能准确识别，且效率远超现有方法。
MoRA: Missing Modality Low-Rank Adaptation for Visual Recognition
- 引入跨模态共享参数，实现图文双向知识迁移。
- 缺失模态下平均性能提升5.24%，推理时间仅为最优方法的25.90%。
- 仅需0.11%可训练参数，适合资源受限场景部署。
预训练视觉语言模型在视觉识别任务中表现优异，但通常假设训练和推理阶段输入模态完整。现实场景中，因隐私、采集难度或资源限制，模态可能缺失。现有方法虽采用提示学习应对，却难以捕捉跨模态关系，且计算开销大。本文提出MoRA，一种参数高效微调方法，显式建模跨模态交互并保持模态特异性适配。MoRA在文本与视觉编码器间引入共享参数，实现双向知识传递；结合模态特异性参数，使模型兼具跨模态关联性与模态内灵活性。在标准基准上的实验表明，MoRA在缺失模态场景下平均性能提升5.24%，推理时间仅为当前最优方法的25.90%，可训练参数仅占全微调的0.11%。
原文摘要 · Abstract (English)
Pre-trained vision language models have shown remarkable performance on visual recognition tasks, but they typically assume the availability of complete multimodal inputs during both training and inference. In real-world scenarios, however, modalities may be missing due to privacy constraints, collection difficulties, or resource limitations. While previous approaches have addressed this challenge using prompt learning techniques, they fail to capture the cross-modal relationships necessary for effective multimodal visual recognition and suffer from inevitable computational overhead. In this paper, we introduce MoRA, a parameter-efficient fine-tuning method that explicitly models cross-modal interactions while maintaining modality-specific adaptations. MoRA introduces modality-common parameters between text and vision encoders, enabling bidirectional knowledge transfer. Additionally, combined with the modality-specific parameters, MoRA allows the backbone model to maintain inter-modality interaction and enable intra-modality flexibility. Extensive experiments on standard benchmarks demonstrate that MoRA achieves an average performance improvement in missing-modality scenarios by 5.24% and uses only 25.90% of the inference time compared to the SOTA method while requiring only 0.11% of trainable parameters compared to full fine-tuning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。