arXiv:2602.21397cs.CVcs.LG2026-02被引 4

用极少量参数实现高效视觉语言模型适配,性能媲美大模型。

MMLoP: Multi-Modal Low-Rank Prompting for Efficient Vision-Language Adaptation

  • 通过低秩分解压缩多模态提示,仅需11.5K可训练参数。
  • 在11个数据集上达到79.7%基类到新类泛化性能。
  • 适合追求高效率与高性能平衡的研究者使用。

提示学习已成为无需修改预训练权重即可适配视觉语言模型(如CLIP)至下游任务的主流范式。尽管在多个Transformer层中同时扩展视觉和文本提示显著提升性能,但其可训练参数量急剧增加,现有最优方法需数百万参数,背离了提示调优的参数高效初衷。本文提出MMLoP(多模态低秩提示),在保持深层多模态提示的同时,仅需11.5K可训练参数,与早期纯文本方法(如CoOp)相当。MMLoP通过低秩分解对每层视觉与文本提示进行参数化,将提示限制在紧凑子空间,实现参数高效并为后续正则化组件提供基础。为缩小与顶尖方法的准确率差距,引入三个互补组件:自调节一致性损失,在特征与逻辑层锚定提示表示于冻结的零样本CLIP特征;均匀漂移修正,消除提示调优引起的全局嵌入偏移,保留类别判别结构;共享上投影,通过共同低秩因子耦合视觉与文本提示,强制跨模态对齐。在三个基准和11个多样化数据集上的大量实验表明,MMLoP实现了优异的准确率-效率权衡,在多数现有方法(包括参数量高出数个数量级的方法)之上表现更优,且在基类到新类泛化上达成79.70%的调和均值。代码已开源。

原文摘要 · Abstract (English)

Prompt learning has become a dominant paradigm for adapting vision-language models (VLMs) such as CLIP to downstream tasks without modifying pretrained weights. While extending prompts to both vision and text encoders across multiple transformer layers significantly boosts performance, it dramatically increases the number of trainable parameters, with state-of-the-art methods requiring millions of parameters and abandoning the parameter efficiency that makes prompt tuning attractive. In this work, we propose MMLoP (Multi-Modal Low-Rank Prompting), a framework that achieves deep multi-modal prompting with only 11.5K trainable parameters, comparable to early text-only methods like CoOp. MMLoP parameterizes vision and text prompts at each transformer layer through a low-rank factorization that constrains prompts to a compact subspace, providing parameter efficiency while motivating the need for our complementary regularization components. To further close the accuracy gap with state-of-the-art methods, we introduce three complementary components: a self-regulating consistency loss that anchors prompted representations to frozen zero-shot CLIP features at both the feature and logit levels, a uniform drift correction that removes the global embedding shift induced by prompt tuning to preserve class-discriminative structure, and a shared up-projection that couples vision and text prompts through a common low-rank factor to enforce cross-modal alignment. Extensive experiments across three benchmarks and 11 diverse datasets demonstrate that MMLoP achieves a highly favorable accuracy-efficiency tradeoff, outperforming the majority of existing methods including those with orders of magnitude more parameters, while achieving a harmonic mean of 79.70\% on base-to-novel generalization. Code is available at https://github.com/sajjad-ucsb/MMLoP.

多模态提示学习低秩

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。