低秩适配让私密图像分类模型训练更高效且保持高精度
Effective Fine-Tuning of Vision Transformers with Low-Rank Adaptation for Privacy-Preserving Image Classification
- 在ViT每层注入可训练低秩矩阵，仅微调少量参数
- 相比全量微调，参数减少90%以上，准确率几乎不变
- 适合资源受限场景的隐私保护图像分类任务
我们提出一种针对隐私保护视觉变换器（ViT）模型的低秩适配方法，可高效冻结预训练ViT权重。该方法在ViT每个层中注入可训练的低秩分解矩阵，且不同于传统方法，不冻结图像块嵌入层。该方法不仅显著减少可训练参数数量，还能维持与全量微调相近的准确率。
原文摘要 · Abstract (English)
We propose a low-rank adaptation method for training privacy-preserving vision transformer (ViT) models that efficiently freezes pre-trained ViT model weights. In the proposed method, trainable rank decomposition matrices are injected into each layer of the ViT architecture, and moreover, the patch embedding layer is not frozen, unlike in the case of the conventional low-rank adaptation methods. The proposed method allows us not only to reduce the number of trainable parameters but to also maintain almost the same accuracy as that of full-time tuning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。