arXiv:2602.10513cs.CVcs.AI2026-02

仅用1%参数实现高效视觉模型适配,超越全量微调和传统方法。

1%>100%: High-Efficiency Visual Adapter with Complex Linear Projection Optimization

  • 设计低秩复数投影适配器,仅引入约1%参数。
  • 理论证明并解决低秩矩阵训练收敛问题,提升稳定性。
  • 在检测、分割等任务上以1%参数超越全量微调,适合部署场景。

视觉基础模型的部署通常依赖高效的适配策略,而传统全量微调成本过高且效率低下。尽管增量微调(delta-tuning)在大语言模型中表现出色,但其优势难以直接迁移至视觉模型微调流程。为突破视觉任务适配效率的瓶颈,本文提出复杂线性投影优化适配器(CoLin)。架构上,设计一种新型低秩复数适配器,仅向主干网络引入约1%参数;效率方面,理论证明低秩复合矩阵在训练中存在严重收敛问题,并通过定制化损失函数加以解决。在目标检测、分割、图像分类及旋转目标检测(遥感场景)上的大量实验表明,CoLin首次以仅1%参数量超越全量微调与经典增量微调方法,为视觉基础模型部署提供了新颖高效的解决方案。代码已开源:https://github.com/DongshuoYin/CoLin。

原文摘要 · Abstract (English)

Deploying vision foundation models typically relies on efficient adaptation strategies, whereas conventional full fine-tuning suffers from prohibitive costs and low efficiency. While delta-tuning has proven effective in boosting the performance and efficiency of LLMs during adaptation, its advantages cannot be directly transferred to the fine-tuning pipeline of vision foundation models. To push the boundaries of adaptation efficiency for vision tasks, we propose an adapter with Complex Linear Projection Optimization (CoLin). For architecture, we design a novel low-rank complex adapter that introduces only about 1% parameters to the backbone. For efficiency, we theoretically prove that low-rank composite matrices suffer from severe convergence issues during training, and address this challenge with a tailored loss. Extensive experiments on object detection, segmentation, image classification, and rotated object detection (remote sensing scenario) demonstrate that CoLin outperforms both full fine-tuning and classical delta-tuning approaches with merely 1% parameters for the first time, providing a novel and efficient solution for deployment of vision foundation models. We release the code on https://github.com/DongshuoYin/CoLin.

视觉适配低秩优化高效微调复数投影

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。