KARST通过多核克罗内克适配提升视觉分类效率,参数少、效果好。
KARST: Multi-Kernel Kronecker Adaptation with Re-Scaling Transmission for Visual Classification
- 用多核克罗内克投影拆分适配矩阵,降低参数依赖
- 引入可学习重缩放因子,更好对齐预训练特征分布
- 轻量高效,适合资源受限场景下的模型微调
在计算机视觉中,微调预训练视觉模型是常见做法。但随着模型规模增大,微调成本显著上升。近期提出的参数高效微调(PEFT)方法通过在预训练骨干网络中引入低秩模块来提升训练效率并减少存储开销。然而,这些方法存在表示能力有限和与预训练中间特征不匹配的问题。为此,本文提出一种新型的多核克罗内克适配与重缩放传输方法(KARST),适用于多种识别任务。其多核设计横向扩展克罗内克投影,并将适配矩阵分解至多个互补空间,降低参数依赖,形成更紧凑的子空间。同时,引入可学习的重缩放因子以更好地对齐预训练特征分布,实现更灵活、均衡的特征融合。大量实验表明,相比其他PEFT方法,KARST在几乎无额外推理开销的前提下表现更优,得益于其重参数化特性。代码已公开于:https://github.com/Lucenova/KARST。
原文摘要 · Abstract (English)
Fine-tuning pre-trained vision models for specific tasks is a common practice in computer vision. However, this process becomes more expensive as models grow larger. Recently, parameter-efficient fine-tuning (PEFT) methods have emerged as a popular solution to improve training efficiency and reduce storage needs by tuning additional low-rank modules within pre-trained backbones. Despite their advantages, they struggle with limited representation capabilities and misalignment with pre-trained intermediate features. To address these issues, we introduce an innovative Multi-Kernel Kronecker Adaptation with Re-Scaling Transmission (KARST) for various recognition tasks. Specifically, its multi-kernel design extends Kronecker projections horizontally and separates adaptation matrices into multiple complementary spaces, reducing parameter dependency and creating more compact subspaces. Besides, it incorporates extra learnable re-scaling factors to better align with pre-trained feature distributions, allowing for more flexible and balanced feature aggregation. Extensive experiments validate that our KARST outperforms other PEFT counterparts with a negligible inference cost due to its re-parameterization characteristics. Code is publicly available at: https://github.com/Lucenova/KARST.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。