通过缓存早期特征提升多模态模型在缺失模态时的鲁棒性。
Buffer replay enhances the robustness of multimodal learning under missing-modality
- 用残差旁路缓存各模态早期特征,深层复用以减少信息丢失
- 分离私有与共享缓冲区,分别保存模态特异和跨模态语义
- 动态初始化机制适配不同缺失场景,增强泛化能力
模态缺失导致多模态模型性能显著下降。现有方法或需高计算成本合成缺失模态,或依赖邻近层特征的提示微调,忽略长距离上下文信息,难以有效应对缺失。为此,我们提出重放提示(REP):(1) 通过残差旁路构建模态级特征缓存,将早期表示重放至深层,缓解网络深度带来的信息损失;(2) 采用私有-共享特征解耦策略,私有缓冲区保留模态特异性信号,共享缓冲区编码跨模态语义;(3) 设计任务感知的动态初始化机制,根据不同缺失条件配置缓冲区,提升稳定性与泛化性。在视觉-语言、视觉-语言-音频及时间多模态基准上的实验表明,REP在单模态与多模态缺失场景下均持续优于已有方法,且参数开销可忽略。结果验证了REP作为轻量高效鲁棒多模态学习范式在挑战性缺失环境中的有效性。
原文摘要 · Abstract (English)
Missing modalities consistently lead to significant performance degradation in multimodal models. Existing approaches either synthesize missing modalities at high computational cost or apply prompt-based fine-tuning that relies only on adjacent-layer features and overlooks long-distance contextual information, which may offer additional tolerance to errors when one or more modalities are missing. To address this, we introduce REplay Prompting (REP): (1) construct modality-wise feature buffers via a residual bypass to cache early-layer representations and replay them in deeper layers, mitigating information loss as network depth increases; (2) employ a private-shared feature decoupling strategy, where private buffers preserve modality-specific signals and shared buffers encode cross-modal semantics; and (3) design a task-aware dynamic initialization mechanism to configure these buffers differently, improving stability and generalization under diverse missing-modality conditions. Experiments on vision-language, vision-language-audio, and temporal multimodal benchmarks demonstrate that REP consistently outperforms prior methods under both single- and multi-modality missing scenarios, while introducing only negligible parameter overhead. These results establish REP as a lightweight and effective paradigm for robust multimodal learning in challenging missing-modality environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。