无需反向传播的视觉语言模型微调方法，提升精度与收敛速度
SharpZO: Hybrid Sharpness-Aware Vision Language Model Prompt Tuning via Forward-Only Passes
- 分两阶段优化：先用尖锐度感知进化策略全局探索，再用稀疏零阶优化局部搜索
- 仅依赖前向传播，实验显示比现有方法平均提升7%准确率
- 适合内存受限的边缘设备部署，尤其适用于无法使用梯度的场景
微调视觉语言模型在下游任务中表现优异，但需要通过反向传播获取梯度，不适用于内存受限的推理型边缘设备。现有无反向传播微调方法多依赖高方差进化策略或零阶优化，性能不佳。本文提出一种混合尖锐度感知零阶优化（SharpZO）方法，通过尖锐度感知预热训练提升零阶微调性能。SharpZO采用两阶段优化：第一阶段使用尖锐度感知进化策略全局探索并平滑损失曲面，生成良好初始化；第二阶段通过稀疏零阶优化进行精细局部搜索。整个过程仅依赖前向传播。理论分析与大量实验表明，该方法显著提升准确率与收敛速度，在CLIP模型上相较当前最优前向仅方法平均提升达7%。
原文摘要 · Abstract (English)
Fine-tuning vision language models (VLMs) has achieved remarkable performance across various downstream tasks; yet, it requires access to model gradients through backpropagation (BP), making them unsuitable for memory-constrained, inference-only edge devices. To address this limitation, previous work has explored various BP-free fine-tuning methods. However, these approaches often rely on high-variance evolutionary strategies (ES) or zeroth-order (ZO) optimization, and often fail to achieve satisfactory performance. In this paper, we propose a hybrid Sharpness-aware Zeroth-order optimization (SharpZO) approach, specifically designed to enhance the performance of ZO VLM fine-tuning via a sharpness-aware warm-up training. SharpZO features a two-stage optimization process: a sharpness-aware ES stage that globally explores and smooths the loss landscape to construct a strong initialization, followed by a fine-grained local search via sparse ZO optimization. The entire optimization relies solely on forward passes. Detailed theoretical analysis and extensive experiments on CLIP models demonstrate that SharpZO significantly improves accuracy and convergence speed, achieving up to 7% average gain over state-of-the-art forward-only methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。