arXiv:2508.11870cs.CVcs.AI2025-08被引 2

用张量环分解让视觉语言模型适配更轻量,参数减少90%仍保持领先性能。

AdaRing: Towards Ultra-Light Vision-Language Adaptation via Cross-Layer Tensor Ring Decomposition

  • 通过跨层张量环分解,将适配器共享核心与层特定切片结合。
  • 在多个任务上实现90%的参数量减少,同时保持最佳性能。
  • 适合资源受限场景下的高效视觉语言模型微调。

基于适配器的微调在高效适配大规模预训练视觉语言模型(VLMs)方面备受关注。该方法仅微调插入的适配器,无需训练原始模型主干。现有方法通过在每个层级集成适配器来提升容量,但面临两大局限:1)因忽略跨层冗余导致压缩率有限;2)同质适配器表示能力受限。本文提出一种基于跨层张量环分解(TRD)的新框架AdaRing,实现超轻量级视觉语言模型微调。通过利用张量级别的低秩特性,将适配器建模为层共享张量核与层特定切片,消除跨层冗余。同时,在泛化感知微调引导下,多样化的秩驱动适配器协同处理需不同表征的任务。实验表明,AdaRing在多个任务上达到当前最优性能,平均训练参数减少90%。

原文摘要 · Abstract (English)

Adapter-based fine-tuning has gained remarkable attention in adapting large pre-trained vision language models (VLMs) for a wide range of downstream tasks efficiently. In this paradigm, only the inserted adapters are fine-tuned, without the need for training the original VLM backbone. Existing works scale adapters by integrating them into every layer of VLMs to increase the capacity of adapters. However, these methods face two primary limitations: 1) limited compression rate due to ignoring cross-layer redundancy, and 2) limited representational capacity across homogeneous adapters. In this paper, we propose a novel vision-language fine-tuning framework based on cross-layer tensor ring decomposition (TRD) with the integration and collaboration of diverse adapters, called AdaRing, achieving ultra-light parameter-efficient adaptation of VLMs on various tasks. To remove the high redundancy that exists among adapters across layers, we exploit the tensor-level low-rankness to formulate adapters as layer-shared tensor cores and layer-specific slices. Moreover, guided by generalization-aware fine-tuning, diverse rank-driven adapters cooperate to handle tasks that require different representations. Our experiments show that the proposed AdaRing achieves the state-of-the-art performance while reducing average training parameters by 90%.

视觉语言轻量化张量分解

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。