arXiv:2505.10088cs.CV2025-05IJCV被引 5

通过可学习的跨模态表示空间,提升视觉语言模型在少样本下的适应能力与泛化性能。

MMRL++: Parameter-Efficient and Interaction-Aware Representation Learning for Vision-Language Models

  • 引入共享可学习的模态无关表示空间,用空间令牌增强图文交互。
  • 在高层编码器插入表示令牌,联合优化类别与表示特征,保持预训练知识。
  • 参数高效设计,支持梯度共享,适合少样本迁移学习场景。

大规模预训练视觉语言模型(VLMs)显著推动了跨任务迁移学习的发展。然而,在少量样本下微调常导致过拟合,削弱其泛化能力。为此,我们提出多模态表示学习(MMRL),构建一个共享、可学习、模态无关的表示空间。该方法将空间令牌投影至文本和图像编码器中作为表示令牌,促进更有效的跨模态交互。不同于以往仅优化类别令牌的方法,MMRL将表示令牌插入更高层编码器——那里任务特异性特征更明显——同时保留底层通用知识。训练时,类别和表示特征联合优化:对表示令牌使用可训练投影层以实现任务适配,而类别令牌的投影层保持冻结以保留预训练知识。为进一步提升泛化性,引入正则项,使类别与文本特征对齐于冻结VLM的零样本特征。推理时,基础任务使用类别与表示特征联合决策,新任务仅使用类别特征以保证更强泛化性。在此基础上,提出参数高效且交互感知的扩展版本MMRL++,显著减少可训练参数,增强表示令牌间的层间交互,使梯度与实例特异性信息更有效传播。在15个数据集上的大量实验表明,MMRL与MMRL++持续优于现有最优方法,实现了任务适应性与泛化能力的优异平衡。

原文摘要 · Abstract (English)

Large-scale pre-trained Vision-Language Models (VLMs) have significantly advanced transfer learning across diverse tasks. However, adapting these models with limited few-shot data often leads to overfitting, undermining their ability to generalize to new tasks. To address this, we propose Multi-Modal Representation Learning (MMRL), which introduces a shared, learnable, modality-agnostic representation space. MMRL generates space tokens projected into both text and image encoders as representation tokens, enabling more effective cross-modal interactions. Unlike prior methods that mainly optimize class token features, MMRL inserts representation tokens into higher encoder layers--where task-specific features are more prominent--while preserving general knowledge in the lower layers. During training, both class and representation features are jointly optimized: a trainable projection layer is applied to representation tokens for task adaptation, while the projection layer for class token remains frozen to retain pre-trained knowledge. To further promote generalization, we introduce a regularization term aligning class and text features with the frozen VLM's zero-shot features. At inference, a decoupling strategy uses both class and representation features for base tasks, but only class features for novel tasks due to their stronger generalization. Building upon this, we propose MMRL++, a parameter-efficient and interaction-aware extension that significantly reduces trainable parameters and enhances intra-modal interactions--particularly across the layers of representation tokens--allowing gradient sharing and instance-specific information to propagate more effectively through the network. Extensive experiments on 15 datasets demonstrate that MMRL and MMRL++ consistently outperform state-of-the-art methods, achieving a strong balance between task-specific adaptation and generalization.

视觉语言模型少样本学习参数效率跨模态交互

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。