让卷积模型高效微调，保留空间结构，效果媲美全量训练。
LoCA: Spatially-Aware Low-Rank Convolutional Adaptation of Vision Foundation Models
- 将卷积核的通道与空间信息分离，用低秩矩阵分别适配。
- 在多个任务上达到顶尖性能，且参数量仅增加1.2%。
- 适合需要高效微调视觉模型的研究者和工程师。
预训练视觉基础模型（VFMs）为多种下游任务提供强大的视觉表征。但其微调面临全量微调成本过高和灾难性遗忘的问题。目前主流的低秩微调（LoRA）方法主要针对由二维矩阵参数化的Transformer自注意力层设计，而卷积核以四维张量耦合空间与通道信息，强行压缩为二维矩阵会破坏其空间拓扑结构。本文提出低秩卷积适应（LoCA），一种面向卷积的参数高效微调框架，通过解耦通道与空间适应，实现对通道间密集混合的低秩建模，并利用奇异值分解（SVD）优化从预训练核中提取的空间基。实验表明，LoCA有效保留了预训练的空间先验，在细粒度分类、领域泛化语义分割及生成基准上均取得竞争性或领先性能。
原文摘要 · Abstract (English)
Pre-trained Vision Foundation Models (VFMs) provide strong visual representations for diverse downstream tasks. The key challenge of VFM adaptation stems from the prohibitive costs of full fine-tuning and catastrophic forgetting. To address this, Low-Rank Adaptation (LoRA) has emerged as the prevailing paradigm for Parameter-Efficient Fine-Tuning (PEFT). However, LoRA is typically designed for transformer self-attention layers parameterized by 2D matrices. Since convolutional kernels inherently couple spatial and channel information within a 4D tensor, forcing them into a monolithic 2D matrix disrupts the inherent spatial topology. In this paper, we propose Low-Rank Convolutional Adaptation (LoCA), a convolution-aware PEFT framework that addresses spatial-channel entanglement by decoupling channel and spatial adaptation. LoCA introduces a low-rank channel adaptation for dense cross-channel mixing and refines spatial bases extracted from pre-trained kernels via Singular Value Decomposition (SVD). Experimental results show that LoCA preserves pre-trained spatial priors and achieves competitive or state-of-the-art performance across fine-grained classification, domain-generalized semantic segmentation, and generative benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。