arXiv:2502.19844cs.CV2025-02CVPR被引 22

用进化算法自动优化视觉分类提示,提升细粒度识别准确率。

ProAPO: Progressively Automatic Prompt Optimization for Visual Classification

  • 基于进化算法逐步优化提示,从通用模板生成类特定描述。
  • 在13个数据集上超越现有文本提示方法,一-shot设置下性能领先。
  • 适合需要低人工干预、高泛化能力的视觉分类场景。

视觉语言模型(VLMs)通过大规模图像-文本对训练,在图像分类任务中取得显著进展,其性能高度依赖提示质量。尽管大型语言模型(LLMs)生成的视觉描述可增强VLM的泛化能力,但类别特定提示可能因LLM幻觉而存在不准确或缺乏区分性。本文旨在以最少监督、无需人工参与的方式,为细粒度类别寻找视觉判别性提示。提出一种基于进化的算法,将任务模板逐步优化为类别特定描述。由于类别特定候选提示的搜索空间呈指数级增长,导致生成成本高、迭代次数多且易过拟合。为此,我们设计了多种基于编辑和进化的操作,仅需一次LLM查询即可生成多样化候选提示;提出两种采样策略,以更好初始点并减少遍历类别,降低迭代成本;同时引入带熵约束的新适应度评分,缓解过拟合问题。在具有挑战性的单样本图像分类设置下,本方法优于现有基于文本提示的方法,并在13个数据集上改进了LLM生成描述方法的效果。此外,验证了最优提示可提升适配器方法性能,并在不同骨干网络间有效迁移。

原文摘要 · Abstract (English)

Vision-language models (VLMs) have made significant progress in image classification by training with large-scale paired image-text data. Their performances largely depend on the prompt quality. While recent methods show that visual descriptions generated by large language models (LLMs) enhance the generalization of VLMs, class-specific prompts may be inaccurate or lack discrimination due to the hallucination in LLMs. In this paper, we aim to find visually discriminative prompts for fine-grained categories with minimal supervision and no human-in-the-loop. An evolution-based algorithm is proposed to progressively optimize language prompts from task-specific templates to class-specific descriptions. Unlike optimizing templates, the search space shows an explosion in class-specific candidate prompts. This increases prompt generation costs, iterative times, and the overfitting problem. To this end, we first introduce several simple yet effective edit-based and evolution-based operations to generate diverse candidate prompts by one-time query of LLMs. Then, two sampling strategies are proposed to find a better initial search point and reduce traversed categories, saving iteration costs. Moreover, we apply a novel fitness score with entropy constraints to mitigate overfitting. In a challenging one-shot image classification setting, our method outperforms existing textual prompt-based methods and improves LLM-generated description methods across 13 datasets. Meanwhile, we demonstrate that our optimal prompts improve adapter-based methods and transfer effectively across different backbones.

视觉分类提示优化进化算法少样本

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。