用进化算法自动设计视觉语言模型适配方法，省去人工调参。
EvoVLMA: Evolutionary Vision-Language Model Adaptation
- 借助大模型生成代码，分两阶段演化优化特征选择与逻辑计算。
- 8次样本分类下准确率提升1.91点，优于人工设计的APE算法。
- 适合想自动化优化模型适配的科研人员和工程师。
预训练视觉-语言模型（VLMs）通过提示调优、适配器等方法在计算机视觉任务（如少样本识别）中得到应用。然而，现有适配方法依赖人工设计，耗时且需丰富经验。受大语言模型代码生成进展启发，本文提出进化视觉-语言模型适配（EvoVLMA）方法，自动搜索免训练的高效适配算法。我们识别出特征选择和逻辑值计算是免训练适配的核心功能，提出一种两阶段、大模型辅助的进化算法，以分治策略有效应对庞大搜索空间。此外，通过低精度代码转换、基于网页的代码执行与过程监控，提升了搜索的稳定性与效率，构建了高效的自动算法设计系统。大量实验表明，EvoVLMA发现的算法在性能上优于以往人工设计的方法。具体而言，在8次样本图像分类设置下，经典APE算法的识别准确率提升1.91个百分点。该研究为预训练多模态模型适配算法的自动化优化开辟了新路径。代码已开源：https://github.com/kding1225/EvoVLMA
原文摘要 · Abstract (English)
Pre-trained Vision-Language Models (VLMs) have been exploited in various Computer Vision tasks (e.g., few-shot recognition) via model adaptation, such as prompt tuning and adapters. However, existing adaptation methods are designed by human experts, requiring significant time cost and experience. Inspired by recent advances in Large Language Models (LLMs) based code generation, we propose an Evolutionary Vision-Language Model Adaptation (EvoVLMA) method to automatically search training-free efficient adaptation algorithms for VLMs. We recognize feature selection and logits computation as the key functions in training-free VLM adaptation, and propose a two-stage LLM-assisted evolutionary algorithm for optimizing these parts in a sequential manner, effectively addressing the challenge posed by the expansive search space through a divide-and-conquer strategy. Besides, to enhance the stability and efficiency of searching process, we propose low-precision code conversion, web based code execution and process monitoring, leading to a highly effective automatic algorithm design system. Extensive experiments demonstrate that the algorithms found by EvoVLMA can obtain promising results compared to previous manually-designed ones. More specifically, in the 8-shot image classification setting, the classical APE algorithm can be improved by 1.91 points in recognition accuracy. This research opens new possibilities for automating the optimization of adaptation algorithms of pre-trained multimodal models. Code is available at: https://github.com/kding1225/EvoVLMA
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。