用跨模态代理标记提升多模态模型在缺失模态时的鲁棒性
Robust Multimodal Learning via Cross-Modal Proxy Tokens
- 引入跨模态代理标记，用可用模态信息模拟缺失模态的类别特征
- 在五大数据集上，不同缺失率下均优于现有方法，完整模态时性能相当
- 仅需低秩适配器，计算开销小，适合部署在资源受限场景
多模态模型在推理时若某一模态缺失，性能常显著下降。为解决此问题，本文提出一种简单而有效的方法：通过跨模态代理标记（CMPTs）增强对缺失模态的鲁棒性，同时在所有模态齐全时保持强性能。该方法利用可用模态的令牌，不依赖显式模态生成或辅助网络，即可近似缺失模态的类别令牌。为高效学习且计算开销最小，采用冻结单模态编码器中的低秩适配器，并联合优化对齐损失与任务特定损失。在五个多模态数据集上的大量实验表明，该方法在多种缺失率下均优于当前最优基线，在完整模态设置下也表现竞争力。整体上，该方法为多模态鲁棒学习提供了灵活高效的解决方案。代码已公开于：https://github.com/CSIPlab/Cross-Modal-Proxy-Tokens。
原文摘要 · Abstract (English)
Multimodal models often experience a significant performance drop when one or more modalities are missing during inference. To address this challenge, we propose a simple yet effective approach that enhances robustness to missing modalities while maintaining strong performance when all modalities are available. Our method introduces cross-modal proxy tokens (CMPTs), which approximate the class token of a missing modality by attending only to the tokens of the available modality without requiring explicit modality generation or auxiliary networks. To efficiently learn these approximations with minimal computational overhead, we employ low-rank adapters in frozen unimodal encoders and jointly optimize an alignment loss with a task-specific loss. Extensive experiments on five multimodal datasets show that our method outperforms state-of-the-art baselines across various missing rates while achieving competitive results in complete-modality settings. Overall, our method offers a flexible and efficient solution for robust multimodal learning. The code for this paper is available at: https://github.com/CSIPlab/Cross-Modal-Proxy-Tokens.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。