让CLIP在测试时自动调用记忆，精准适配新图像
MINT: Memory-Infused Prompt Tuning at Test-time for CLIP
- 用可学习的提示记忆库存储见过的样本特征
- 根据图像层次特征动态组合提示，实现细粒度上下文引导
- 无需重训练或源数据，适合快速部署于分布漂移场景
视觉-语言预训练模型（VLM）在测试阶段面对数据分布偏移时的泛化能力仍面临挑战。现有测试时自适应（TTA）方法未能充分挖掘模型内部知识，尤其难以动态适应复杂层级的视觉语义信息。本文提出内存注入提示调优（MINT），受人类联想记忆理论启发，引入记忆提示库（MPB），存储可学习的键值提示对，作为过往样本的记忆。测试时，通过测试图像的层次化视觉特征检索相关提示对，动态构建关联提示，并注入图像编码器，实现细粒度、定制化的视觉上下文引导。同时结合可学习文本提示，使MINT在无需源数据或重训练的情况下，快速精确地完成VLM测试时自适应。代码已开源。
原文摘要 · Abstract (English)
Improving the generalization ability of Vision-Language Pre-trained Models (VLMs) under test-time data distribution shifts remains a critical challenge. The existing Test-Time Adaptation (TTA) methods fall short in fully leveraging the model's internal knowledge, particularly in dynamically adapting to complex and hierarchical visual semantic information. In this paper, we propose Memory-Infused Prompt Tuning (MINT), a novel framework to address this issue. Inspired by human associative memory theory, MINT introduces a Memory Prompt Bank (MPB), which stores learnable key-value prompt pairs that work as a memory of previously seen samples. During the test time, relevant prompt pairs in the MPB are retrieved by the hierarchical visual features of test images to dynamically assemble Associative Prompts. The associative prompts are then injected into the image encoder for fine-grained, customized visual contextual guidance. MINT also utilizes learnable text prompts. MINT thus enables rapid, precise VLM adaptation at test time by leveraging this MPB-acquired memory, without source data or retraining. The code is available at https://github.com/Jamieyi2004/MINT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。