针对视觉语言模型的图像编码器，提出高效低秩更新方法。
Efficient Few-Shot Continual Learning in Vision-Language Models
- 通过结构化局部参数更新，精准修正旧数据错误
- 计算开销降低25倍以上，性能不下降
- 适合资源受限场景下的持续学习任务
视觉语言模型（VLMs）在视觉问答和图像描述等任务中表现优异。然而，其性能常受限于预训练图像编码器（如CLIP），导致图像理解错误。现实应用中，模型需持续适应新出现的少量数据。为此，我们提出LoRSU（低秩自适应与结构化更新），一种高效且鲁棒的方法，用于在VLM中选择性更新图像编码器。LoRSU引入结构化、局部化的参数更新机制，有效修正以往易出错数据的表现，同时保持模型整体稳健性。该方法基于理论洞察，仅更新最关键参数，实现显著资源效率提升。实验表明，在少样本持续学习设置下，LoRSU在VQA任务上的计算开销较完整模型更新降低超过25倍，性能无损。结果验证了其可扩展性、高效性与有效性，是资源受限环境下图像编码器适配的有力方案。
原文摘要 · Abstract (English)
Vision-language models (VLMs) excel in tasks such as visual question answering and image captioning. However, VLMs are often limited by their use of pretrained image encoders, like CLIP, leading to image understanding errors that hinder overall performance. On top of that, real-world applications often require the model to be continuously adapted as new and often limited data continuously arrive. To address this, we propose LoRSU (Low-Rank Adaptation with Structured Updates), a robust and computationally efficient method for selectively updating image encoders within VLMs. LoRSU introduces structured and localized parameter updates, effectively correcting performance on previously error-prone data while preserving the model's general robustness. Our approach leverages theoretical insights to identify and update only the most critical parameters, achieving significant resource efficiency. Specifically, we demonstrate that LoRSU reduces computational overhead by over 25x compared to full VLM updates, without sacrificing performance. Experimental results on VQA tasks in the few-shot continual learning setting, validate LoRSU's scalability, efficiency, and effectiveness, making it a compelling solution for image encoder adaptation in resource-constrained environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。