用轻量态空间模块高效微调大模型，省参数还提性能。
Selective State Space Memory for Large Vision-Language Models
- 在视觉语言模型中插入轻量Mamba模块，捕捉长程依赖。
- 仅更新少量参数，实现媲美顶尖模型的多任务表现。
- 适合资源有限但需快速适配新场景的研究与应用。
大型视觉语言模型（LVLMs）在多种多模态任务中表现出色。然而，针对特定领域进行微调仍面临计算成本高的挑战。本文提出状态空间记忆集成（SSMI），一种高效的LVLM微调方法。通过将基于Mamba的轻量级状态空间模块嵌入LVLM架构，SSMI能有效捕捉长程依赖，并注入任务相关的视觉与序列模式。与传统微调方法不同，SSMI仅需更新模型极小部分参数，具有高计算效率与可扩展性。在COCO Captioning、VQA和Flickr30k等基准数据集上的实验表明，SSMI在保持鲁棒性和泛化能力的同时，达到当前最优性能。全面分析进一步验证了其在效率、适应性与可解释性方面的优势，为大规模视觉语言模型的微调提供了有力解决方案。
原文摘要 · Abstract (English)
Large Vision-Language Models (LVLMs) have demonstrated remarkable performance across a wide range of multimodal tasks. However, fine-tuning these models for domain-specific applications remains a computationally intensive challenge. This paper introduces State Space Memory Integration (SSMI), a novel approach for efficient fine-tuning of LVLMs. By integrating lightweight Mamba-based state space modules into the LVLM architecture, SSMI captures long-range dependencies and injects task-specific visual and sequential patterns effectively. Unlike traditional fine-tuning methods, SSMI requires only a fraction of the model's parameters to be updated, making it computationally efficient and scalable. Experiments on benchmark datasets, including COCO Captioning, VQA, and Flickr30k, demonstrate that SSMI achieves state-of-the-art performance while maintaining robustness and generalization capabilities. Comprehensive analysis further validates the advantages of SSMI in terms of efficiency, adaptability, and interpretability, positioning it as a compelling solution for fine-tuning large-scale vision-language models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。