用损失值自动选最优视觉提示，提升大模型图像理解能力
AutoV: Loss-Oriented Ranking for Visual Prompt Retrieval in LVLMs
- 根据预训练模型的预测损失排序候选视觉提示
- 在VizWiz和MMMU上分别提升10.2%和3.8%
- 无需人工标注，适合各类视觉语言模型增强
受大型语言模型中文本提示启发，视觉提示被用于增强大视觉语言模型（LVLM）的感知能力。然而，单一视觉提示设计下性能容易饱和，进一步提示工程效果有限。为此，我们从提示工程转向提示检索，提出AutoV——一种轻量级实例自适应视觉提示识别框架。给定输入图像和文本查询，AutoV可自动从多样化候选池中定位最适配的视觉提示。训练该检索框架需提示级监督，但提示质量本身模糊且难以可靠评估，即使对人类亦然。为实现自动监督，我们利用预训练LVLM评估视觉提示并依据其预测损失打标。以损失导向排序作为稳健训练信号，AutoV学习为每个实例检索与查询相关的最优提示，无需人工标注。实验表明，AutoV在图像理解、描述生成、定位和分类任务中均提升了多种LVLM的表现。例如，其使LLaVA-OV在VizWiz上提升10.2%，使Qwen2.5-VL在MMMU上提升3.8%。
原文摘要 · Abstract (English)
Inspired by text prompts in large language models, visual prompts have been explored to enhance the perceptual capabilities of large vision-language models (LVLMs). However, performance tends to saturate under single visual prompt designs, making further prompt engineering increasingly ineffective. To address this limitation, we shift from prompt engineering to prompt retrieval and propose AutoV, a lightweight framework for instance-adaptive visual prompt identification. Given an input image and a textual query, AutoV automatically locates the most suitable visual prompt from a diverse candidate pool. Training such a retrieval framework requires prompt-level supervision, yet prompt quality is inherently ambiguous and difficult to assess reliably, even for humans. To enable automatic supervision, we evaluate visual prompts using a pre-trained LVLM and label them according to their prediction losses. Using the loss-oriented ranking as a robust training signal, AutoV learns to retrieve the query-aware optimal prompt for each instance without manual annotation. Experiments indicate that AutoV enhances the performance of various LVLMs on image understanding, captioning, grounding, and classification tasks. For example, AutoV improves LLaVA-OV by $\textbf{10.2}\%$ on VizWiz and boosts Qwen2.5-VL by $\textbf{3.8}\%$ on MMMU, respectively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。