在2GB显存下，用自适应检查点提升视觉模型微调效率
Efficient PEFT Methods with Adaptive Checkpointing for Vision Models and VLMs on Resource Constrained Consumer-GPUs
- 提出自适应检查点算法，动态分配内存以降低峰值显存
- QLoRA和BitFit在低精度下能耗降低20-30%，损失1-2%准确率
- DINOv2模型在资源受限场景下表现优于微调模型
现代预训练视觉模型虽精度高，但微调需大量显存，难以部署于边缘设备。本文在2GB显存限制下，对比了五种参数高效微调方法（全量微调、LoRA、AdaLoRA、QLoRA、BitFit）在Transformer（ViT-Small、TinyViT）与Mamba架构（Vim-Small、MambaVision-T）上的表现，并结合三种梯度检查点策略（无、静态、新提出的内存预算感知自适应算法），评估三类基础模型：零样本对比视觉语言模型（OpenCLIP、SigLIP）、轻量级自监督视觉主干（DINOv2）以及基于提示的自回归视觉语言模型（PaliGemma、MobileVLM、SmolVLM）。在CIFAR-100和DTD数据集上测试了准确率、训练时间、能耗及NetScore多目标指标，后者扩展了两个面向部署的变体。结果表明，QLoRA和BitFit在1-2%准确率损失下实现20-30%能耗降低；自适应算法使峰值显存减少43-79%，能耗增加9-30%。DINOv2在CIFAR-100上准确率达0.917，高于微调模型的0.897，且能耗极低，而小型自回归VLM仍不具竞争力。
原文摘要 · Abstract (English)
Modern pretrained vision models achieve strong accuracy but demand substantial GPU memory for fine-tuning, making edge deployment impractical. This paper compares five parameter-efficient fine-tuning (PEFT) methods (Full FT, LoRA, AdaLoRA, QLoRA, BitFit) on Transformers- (ViT-Small, TinyViT) and Mamba-based vision backbones (Vim-Small, MambaVision-T) under an on-device VRAM budget (e.g., 2 GB), together with three gradient-checkpointing strategies (none, static, and a proposed memory-budget-aware adaptive algorithm); and we evaluate three families of foundation-model baselines: zero-shot contrastive vision language models (OpenCLIP, SigLIP), self-supervised vision backbones with lightweight evaluation protocols (DINOv2), and autoregressive VLMs for prompt-based classification (PaliGemma, MobileVLM, SmolVLM). Experiments on CIFAR-100 and DTD report accuracy, training time, energy, and the NetScore family of multi-objective metrics, which we extend with two deployment-aware variants. QLoRA and BitFit cut energy 20-30% at a 1-2% accuracy cost; the adaptive algorithm reduces peak memory 43-79% with 9-30% energy overhead. DINOv2 surpasses fine-tuned models on CIFAR-100 (0.917 vs. 0.897) at a fraction of the energy, while small autoregressive VLMs remain uncompetitive.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。