轻量级方法让大模型高效适配下游任务,精度媲美全量微调。
SIGMA: Bridging Structural and Distributional Gaps for Vision Foundation Model Adaptation

- 通过多尺度融合与语义调制,同时解决结构与分布差异问题。
- 仅需1.72%可训练参数,实现比现有方法更优的密集预测性能。
- 适合资源有限场景下快速部署视觉大模型,尤其适合密集任务。
视觉基础模型(VFMs)展现出强大的表征能力,但通过全量微调适配下游任务会带来高昂的计算与存储开销。参数高效微调(PEFT)作为替代方案,旨在以极低训练成本实现与全量微调相当的性能。然而,在密集预测任务中应用PEFT仍面临结构与分布差距的挑战。为此,本文提出一种新型轻量级PEFT方法SIGMA,包含两个模块:尺度自适应融合与语义调制。其中,尺度自适应融合模块增强多粒度视觉信息提取,以弥合结构差距;语义调制模块对融合特征进行全局特征对齐,进一步消除分布差距。该设计实现了统一的空间与分布适应,仅需1.72%的可训练参数。在多种下游密集任务及多个VFM骨干网络上进行的全面实验表明,SIGMA持续优于当前最先进的PEFT方法。
原文摘要 · Abstract (English)
Vision Foundation Models (VFMs) have demonstrated impressive representational capabilities. However, adapting them to downstream tasks via full fine-tuning incurs prohibitive computational and storage overhead. Parameter-Efficient Fine-Tuning (PEFT) has emerged as a compelling alternative, aiming to achieve performance parity with full fine-tuning at minimal training costs. Nonetheless, applying PEFT to VFMs for dense prediction tasks remains challenging due to the structural and distributional gaps. To bridge these gaps, we propose \textbf{S}cale-\textbf{I}ntegrated \textbf{G}lobal \textbf{M}odulation \textbf{A}dapter (\textbf{SIGMA}), a novel lightweight PEFT method, which consists of two modules: scale-adaptive fusion and semantic modulation. Specifically, the scale-adaptive fusion module is utilized to bridge structural gaps by enhancing the extraction of multi-granularity visual information. Furthermore, SIGMA introduces semantic modulation on the fusion features to perform global feature alignment to further eliminate the distribution gap. This design facilitates unified spatial and distributional adaptation, requiring only 1.72\% trainable parameters relative to the VFM backbone. Comprehensive experiments across various downstream dense tasks and multiple VFM backbones demonstrate that SIGMA achieves consistent and superior performance over state-of-the-art PEFT methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。