arXiv:2605.19301cs.CV2026-05

提出iGSP框架,让视觉语言模型持续学习时更省参数、不丢知识。

iGSP:Implicit Gradient Subspace Projection for Efficient Continual Learning of Vision-Language Models

论文配图:iGSP:Implicit Gradient Subspace Projection for Efficient Continual Learning of Vision-Language Models
图 1 · 摘自论文原文
  • 通过隐式梯度子空间投影,让新任务共享旧任务的优化方向。
  • 在MTIL上训练参数减少42.7%,最终总参数降86.9%且准确率领先。
  • 适合需要长期更新的视觉语言模型应用,如多任务智能助手。

视觉语言模型需高效适应持续出现的新下游任务。虽然参数高效微调可缓解灾难性遗忘,但为每项任务分配独立模块会导致参数爆炸;而现有基于相似性的共享机制错误地将表面视觉相似性等同于深层对齐一致性,导致视觉相似但逻辑不同的任务间产生严重负迁移,并未能利用视觉差异任务间的对齐复用。我们指出,对齐共享本质上是共享低秩子空间中优化轨迹重叠的几何问题。基于此,提出iGSP框架,通过隐式梯度子空间投影实现高效适应。利用MoE路由早期收敛建立子空间基底,iGSP分两阶段:第一阶段(子空间识别)通过基底扩展引入候选专家,施加新型子空间约束正则化,隐式将新任务梯度投影至历史子空间,并以路由概率为梯度流指标精确剪枝冗余维度,最大化知识复用;第二阶段(正交子空间微调)固定结构基底,移除正则化,快速拟合任务特定残差损失。在MTIL基准上的大量实验表明,iGSP达到当前最优准确率,同时显著提升训练效率,相比当前SOTA方法平均可训练参数减少42.7%,最终总参数减少86.9%。代码已开源。

原文摘要 · Abstract (English)

Vision-Language Models require efficient adaptation to continually emerging downstream tasks. While Parameter-Efficient Fine-Tuning mitigates catastrophic forgetting, assigning isolated modules per task leads to parameter explosion. Conversely, recent similarity-driven sharing mechanisms falsely equate superficial visual similarity with underlying alignment consistency. This fundamental mismatch triggers severe negative transfer between visually similar but logically distinct tasks and fails to exploit alignment reuse across visually diverse ones. We argue thatalignment sharing is fundamentally a geometric problem of overlapping optimization trajectories within shared low-rank subspaces. Grounded in this insight, we propose iGSP, a novel framework that achieves efficient adaptation via implicit gradient subspace projection. Leveraging the early convergence of MoE routers to establish the subspace basis, iGSP bifurcates the adaptation process into two phases. First, the Subspace Identification phase introduces candidate experts via basis pre-expansion, applies a novel subspace-constrained regularization to implicitly project new task gradients onto the historical subspace, and precisely prunes redundant dimensions by treating routing probabilities as gradient flow indicators, ultimately to maximize knowledge reuse. Second, the Orthogonal Subspace Fine-Tuning phase fixes this structural basis and removes the regularization to rapidly fit the task-specific residual loss. Extensive experiments on the MTIL benchmark demonstrate that iGSP achieves state-of-the-art accuracy while significantly improving training efficiency, reducing the average trainable parameters by 42.7\% compared to current SOTA methods, and decreasing the final total parameters by 86.9\% relative to counterparts. The source code is available at https://github.com/GeoX-Lab/iGSP.

持续学习视觉语言模型参数效率子空间投影

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。