发现多数样本用同一提示即可达最优，提出任务级提示降低推理开销。
Exploring Task-Level Optimal Prompts for Visual In-Context Learning
- 基于任务级提示，统一为所有样本使用相同最优提示。
- 实验表明该方法在极低计算成本下达到最佳视觉上下文学习性能。
- 适合追求高效部署的视觉基础模型应用者。
近年来，视觉基础模型（VFMs）的发展使视觉上下文学习（VICL）在多数场景下成为优于微调的方案。与重新训练或微调模型不同，VICL无需修改模型权重或结构，仅需包含示范的提示即可指导模型完成任务。然而，当前寻找每个测试样本最优提示存在巨大计算开销，因确定构建提示所用示范内容极为耗时。本文发现一个反直觉现象：大多数测试样本在相同提示下即可达到最优性能，而逐样本搜索提示仅增加时间开销却得到完全相同的提示。为此，我们提出任务级提示策略，在推理阶段大幅降低提示搜索成本，并引入两种高效且有效的任务级提示搜索方法。大量实验结果表明，本方法可识别近似最优提示，在先前工作从未达到的极低开销下实现最佳VICL性能。
原文摘要 · Abstract (English)
With the development of Vision Foundation Models (VFMs) in recent years, Visual In-Context Learning (VICL) has become a better choice compared to modifying models in most scenarios. Different from retraining or fine-tuning model, VICL does not require modifications to the model's weights or architecture, and only needs a prompt with demonstrations to teach VFM how to solve tasks. Currently, significant computational cost for finding optimal prompts for every test sample hinders the deployment of VICL, as determining which demonstrations to use for constructing prompts is very costly. In this paper, however, we find a counterintuitive phenomenon that most test samples actually achieve optimal performance under the same prompts, and searching for sample-level prompts only costs more time but results in completely identical prompts. Therefore, we propose task-level prompting to reduce the cost of searching for prompts during the inference stage and introduce two time-saving yet effective task-level prompt search strategies. Extensive experimental results show that our proposed method can identify near-optimal prompts and reach the best VICL performance with a minimal cost that prior work has never achieved.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。