不改权重，用门控重连模型内部结构，让视觉语言模型更聪明。
Rethinking Fine-Tuning: Unlocking Hidden Capabilities in Vision-Language Models
- 用可学习门控分数重排模型内部连接，替代传统参数更新。
- 在多个VLM上超越LoRA和全量微调，性能更高且不改动主干。
- 适合想高效适配大模型又不想烧钱的开发者和研究者。
视觉语言模型（VLM）的微调研究，如参数高效微调（PEFT）中的低秩适应（LoRA），取得了显著进展。然而，大多数方法依赖显式权重更新，忽略了预训练模型中已编码的丰富表征结构，这些结构长期被忽视。近期研究表明，掩码微调（MFT）作为语言模型的高效后训练范式具有强大潜力。MFT不更新权重，而是为每个权重分配可学习的门控分数，使模型能重新组织其内部子网络以适应下游任务。本文从结构重参数化视角重新思考VLM的微调，将MFT应用于不同语言主干的VLM的语言与投影组件，并与强基线PEFT方法对比。实验表明，MFT始终优于LoRA变体，甚至超过全量微调，在冻结主干的前提下实现高精度表现。研究揭示：有效适应不仅来自权重更新，也源于对模型已有知识间连接关系的重构。代码已公开：https://github.com/Ming-K9/MFT-VLM
原文摘要 · Abstract (English)
Explorations in fine-tuning Vision-Language Models (VLMs), such as Low-Rank Adaptation (LoRA) from Parameter Efficient Fine-Tuning (PEFT), have made impressive progress. However, most approaches rely on explicit weight updates, overlooking the extensive representational structures already encoded in pre-trained models that remain underutilized. Recent works have demonstrated that Mask Fine-Tuning (MFT) can be a powerful and efficient post-training paradigm for language models. Instead of updating weights, MFT assigns learnable gating scores to each weight, allowing the model to reorganize its internal subnetworks for downstream task adaptation. In this paper, we rethink fine-tuning for VLMs from a structural reparameterization perspective grounded in MFT. We apply MFT to the language and projector components of VLMs with different language backbones and compare against strong PEFT baselines. Experiments show that MFT consistently surpasses LoRA variants and even full fine-tuning, achieving high performance without altering the frozen backbone. Our findings reveal that effective adaptation can emerge not only from updating weights but also from reestablishing connections among the model's existing knowledge. Code available at: https://github.com/Ming-K9/MFT-VLM
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。