让视觉语言模型跨模态交互更直接,提升小样本任务表现
MAIL++: Multi-Modal Bi-directional Agent Layer for Vision-Language Models

- 将跨模态耦合嵌入模型核心计算层,实现细粒度交互
- 在少样本图像分类与跨域检索上超越现有最先进方法
- 适合追求高效微调且需强跨模态理解的场景
将大型视觉语言模型(如CLIP)适配到下游任务仍具挑战性,全量微调计算成本高且在低数据情况下易过拟合。参数高效微调(PEFT)通过轻量级提示或适配器模块缓解此问题,而跨模态耦合已被证明能增强视觉与语言间的互动。然而,现有机制多依赖外部辅助模块,导致交互间接、粗粒度,且结构上与原模型解耦,限制表征能力。本文提出多模态交互代理层(MAIL),将跨模态耦合直接嵌入VLM的核心计算模块中。MAIL冻结主干网络,在如LayerNorm等核心模块后插入轻量代理层,以近似全量微调带来的参数更新。为在该层级实现视觉与文本流的耦合,引入基于瓶颈的文本到图像桥接机制,联合优化跨模态对应代理层,协调相应计算模块的适应。进一步提出MAIL++,通过元代理层、元文本桥和元图像桥实现双向跨模态交换。推理时,所有代理层可重参数化为冻结主干,保持原始计算效率。大量实验表明,MAIL与MAIL++在少样本图像分类和少样本通用跨域检索任务上持续优于当前最优的PEFT方法。
原文摘要 · Abstract (English)
Adapting large vision-language models (VLMs) such as CLIP to downstream tasks remains challenging, as full fine-tuning is computationally prohibitive and prone to overfitting in low-data regimes. Parameter-efficient fine-tuning (PEFT) alleviates these issues with lightweight prompt- or adapter-based modules, and cross-modal coupling has proven especially effective by strengthening interactions between vision and language. However, existing coupling mechanisms predominantly rely on external auxiliary modules, leading to indirect, coarse-grained interactions that are structurally decoupled from the original VLM and thus limit representational expressiveness. In this paper, we propose Multi-Modal Interactive Agent Layer (MAIL), a PEFT paradigm that embeds cross-modal coupling directly into the intrinsic computation modules of VLMs. MAIL freezes the backbone and inserts lightweight agent layers after core modules, such as LayerNorm, to approximate the parameter updates induced by full fine-tuning. To couple visual and textual streams at this level, we introduce a bottleneck-based text-to-image bridge that jointly optimizes paired agent layers across modalities, coordinating the adaptation of corresponding computation modules. We further present MAIL++, which enables bidirectional cross-modal exchange through a meta agent layer, a meta-text bridge, and a meta-image bridge. At inference time, all agent layers are re-parameterized into the frozen backbone, preserving the original computational efficiency. Extensive experiments on few-shot image classification and few-shot universal cross-domain retrieval demonstrate that MAIL and MAIL++ consistently outperform state-of-the-art PEFT methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。