用轻量适配器实现精准视觉定位,仅1.4%参数即可超越全量微调。
MaPPER: Multimodal Prior-guided Parameter Efficient Tuning for Referring Expression Comprehension
- 设计动态先验适配器与局部卷积模块,增强跨模态对齐与局部感知。
- 在三个基准上准确率超全微调和现有轻量方法,仅需1.41%可训练参数。
- 适合资源受限场景下的视觉语言理解任务,如移动端或实时应用。
指代表达理解(REC)旨在通过自然语言定位特定视觉区域,高度依赖多模态对齐。现有方法多采用全量微调预训练模型以迁移视觉与语言知识,但此举不仅破坏预训练中蕴含的丰富先验知识,还带来巨大计算开销。受参数高效迁移学习(PETL)启发,本文提出多模态先验引导的参数高效微调框架MaPPER。其核心包含:由对齐先验驱动的动态先验适配器、用于提取精确局部语义的局部卷积适配器,以及先验引导文本模块以强化跨模态对齐。在三个主流基准上的实验表明,MaPPER在仅使用1.41%可训练骨干参数的情况下,达到优于全微调及其他PETL方法的最佳精度。代码已开源。
原文摘要 · Abstract (English)
Referring Expression Comprehension (REC), which aims to ground a local visual region via natural language, is a task that heavily relies on multimodal alignment. Most existing methods utilize powerful pre-trained models to transfer visual/linguistic knowledge by full fine-tuning. However, full fine-tuning the entire backbone not only breaks the rich prior knowledge embedded in the pre-training, but also incurs significant computational costs. Motivated by the recent emergence of Parameter-Efficient Transfer Learning (PETL) methods, we aim to solve the REC task in an effective and efficient manner. Directly applying these PETL methods to the REC task is inappropriate, as they lack the specific-domain abilities for precise local visual perception and visual-language alignment. Therefore, we propose a novel framework of Multimodal Prior-guided Parameter Efficient Tuning, namely MaPPER. Specifically, MaPPER comprises Dynamic Prior Adapters guided by an aligned prior, and Local Convolution Adapters to extract precise local semantics for better visual perception. Moreover, the Prior-Guided Text module is proposed to further utilize the prior for facilitating the cross-modal alignment. Experimental results on three widely-used benchmarks demonstrate that MaPPER achieves the best accuracy compared to the full fine-tuning and other PETL methods with only 1.41% tunable backbone parameters. Our code is available at https://github.com/liuting20/MaPPER.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。