arXiv:2512.24603cs.CV2025-12

通过共享空间与多样性正则,提升视觉Transformer微调的效率与性能。

Collaborative Low-Rank Adaptation for Pre-Trained Vision Transformers

  • 多个低秩模块共享投影空间,协同构建参数高效表示。
  • 在图像与点云数据上,性能优于当前最优方法,点云分析所需计算量最少。
  • 适合追求高效率与高性能平衡的视觉模型微调场景。

低秩适配(LoRA)在微调预训练视觉Transformer方面取得了显著成功。现有方法主要关注更高效的参数策略或更强的表征学习方案,但往往在性能与参数效率之间难以兼顾。为此,本文提出一种新型微调方法——协作低秩适配(CLoRA),包含基空间共享和样本无关多样性增强(SADE)两个组件。为在保持参数效率的同时扩展低秩模块(LRM)的学习能力,基空间共享使所有LRM共享一组上下投影空间;各低秩矩阵从共享空间中协同构建每个LRM。由于这些矩阵提取的表征可能含冗余信息,因此引入SADE通过正则化降低它们之间的相似性,以促进训练过程中的多样性。我们在广泛使用的图像与点云数据集上进行了大量实验,结果表明,相比当前最优方法,CLoRA在学习性能与参数效率之间取得更好平衡,且在点云分析任务中所需的GFLOPs最少。

原文摘要 · Abstract (English)

Low-rank adaptation (LoRA) has achieved remarkable success in fine-tuning pre-trained vision transformers for various downstream tasks. Existing studies mainly focus on exploring more parameter-efficient strategies or more effective representation learning schemes. However, these methods either sacrifice fine-tuning performance or introduce excessive trainable parameters, failing to strike a balance between learning performance and parameter efficiency. To address this problem, we propose a novel tuning method named collaborative low-rank adaptation (CLoRA) in this paper. CLoRA consists of base-space sharing and sample-agnostic diversity enhancement (SADE) components. To maintain parameter efficiency while expanding the learning capacity of low-rank modules (LRMs), base-space sharing allows all LRMs to share a set of down/up-projection spaces. In CLoRA, the low-rank matrices obtained from the shared spaces collaboratively construct each LRM. Since the representations extracted by these matrices may contain redundant information, SADE is employed to regularize the similarities among them to encourage diverse representations in the training process. We conduct extensive experiments on widely used image and point cloud datasets to evaluate the performance of CLoRA. Experimental results demonstrate that CLoRA strikes a better balance between learning performance and parameter efficiency, while requiring the fewest GFLOPs for point cloud analysis, compared with the state-of-the-art methods.

视觉Transformer低秩适配模型压缩点云分析

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。