arXiv:2512.06811cs.CVcs.AI2025-12AAAI被引 1

用重建机制平衡视觉语言模型的通用性与任务适配,提升少样本表现。

RMAdapter: Reconstruction-based Multi-Modal Adapter for Vision-Language Models

  • 双分支结构:一边注入任务知识,一边重建特征保持通用性。
  • 少样本场景下超越现有方法,三项评估指标均领先。
  • 轻量设计适合部署,无需数据增强或重复提示。

预训练的视觉语言模型(如CLIP)已成为多模态迁移学习的重要工具。然而,在少样本场景下微调这些模型时,如何在任务特异性适应与模型泛化能力之间取得平衡仍具挑战。当前研究多集中于基于提示的适配方法,而基于适配器的方法则未被充分探索,且性能存在明显差距。为此,本文提出一种基于重建的多模态适配器(RMAdapter),采用双分支架构:(1)适配分支通过参数高效微调注入任务特定知识;(2)重建分支通过将隐空间特征重建回原始空间来保留通用知识。该设计实现了通用知识与任务知识的动态平衡。尽管引入了额外的重建分支,但通过在每层局部计算重建损失并共享投影模块,整体计算开销极小。同时引入一致性约束,以更好调控可区分性与泛化性的权衡。我们在三个代表性任务上全面评估:新类别泛化、新目标数据集泛化和领域泛化。不依赖数据增强或重复提示设计,RMAdapter在所有评估指标上均持续优于现有先进方法。

原文摘要 · Abstract (English)

Pre-trained Vision-Language Models (VLMs), \textit{e.g.} CLIP, have become essential tools in multimodal transfer learning. However, fine-tuning VLMs in few-shot scenarios poses significant challenges in balancing task-specific adaptation and generalization in the obtained model. Meanwhile, current researches have predominantly focused on prompt-based adaptation methods, leaving adapter-based approaches underexplored and revealing notable performance gaps. To address these challenges, we introduce a novel Reconstruction-based Multimodal Adapter (RMAdapter), which leverages a dual-branch architecture. Unlike conventional single-branch adapters, RMAdapter consists of: (1) an adaptation branch that injects task-specific knowledge through parameter-efficient fine-tuning, and (2) a reconstruction branch that preserves general knowledge by reconstructing latent space features back into the original feature space. This design facilitates a dynamic balance between general and task-specific knowledge. Importantly, although RMAdapter introduces an additional reconstruction branch, it is carefully optimized to remain lightweight. By computing reconstruction loss locally at each layer and sharing projection modules, the overall computational overhead is kept minimal. A consistency constraint is also incorporated to better regulate the trade-off between discriminability and generalization. We comprehensively evaluate the effectiveness of RMAdapter on three representative tasks: generalization to new categories, generalization to new target datasets, and domain generalization. Without relying on data augmentation or duplicate prompt designs, our RMAdapter consistently outperforms state-of-the-art approaches across all evaluation metrics.

多模态适配器少样本知识平衡

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。