通过可学习扰动优化视觉上下文提示，提升模型零样本泛化能力
E-InMeMo: Enhanced Prompting for Visual In-Context Learning
- 在输入图像对中加入可学习扰动，动态增强提示信息
- 在分割和检测任务上分别提升mIoU 7.99和17.04
- 轻量级方案，适合快速适配新视觉任务的场景
大规模模型因在多样化任务中具备强泛化能力已成为标准。在上下文学习（ICL）中，无需修改参数即可通过任务特定提示调用这些模型，该范式正被广泛引入计算机视觉领域。视觉ICL中，模型接收一个输入-输出图像对（即上下文对）与查询图像，以示例期望输出。然而，其性能高度依赖提示质量。为此，我们提出增强指令更多（E-InMeMo），将可学习扰动融入上下文对以优化提示。在标准视觉任务上的大量实验表明，相比无可学习提示的基线，E-InMeMo在前景分割任务上提升mIoU 7.99，单目标检测任务上提升17.04。结果证明E-InMeMo是一种轻量但高效的视觉ICL增强策略。代码已公开于：https://github.com/Jackieam/E-InMeMo
原文摘要 · Abstract (English)
Large-scale models trained on extensive datasets have become the standard due to their strong generalizability across diverse tasks. In-context learning (ICL), widely used in natural language processing, leverages these models by providing task-specific prompts without modifying their parameters. This paradigm is increasingly being adapted for computer vision, where models receive an input-output image pair, known as an in-context pair, alongside a query image to illustrate the desired output. However, the success of visual ICL largely hinges on the quality of these prompts. To address this, we propose Enhanced Instruct Me More (E-InMeMo), a novel approach that incorporates learnable perturbations into in-context pairs to optimize prompting. Through extensive experiments on standard vision tasks, E-InMeMo demonstrates superior performance over existing state-of-the-art methods. Notably, it improves mIoU scores by 7.99 for foreground segmentation and by 17.04 for single object detection when compared to the baseline without learnable prompts. These results highlight E-InMeMo as a lightweight yet effective strategy for enhancing visual ICL. Code is publicly available at: https://github.com/Jackieam/E-InMeMo
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。