动态调整提示词，让视觉语言模型推理更准且不跑偏。
DynaPrompt: Dynamic Test-Time Prompt Tuning
- 根据预测熵和概率差动态选提示，提升相关样本利用效率。
- 在14个数据集上显著优于传统测试时提示调优方法。
- 适合追求零样本泛化性能的视觉语言模型研究者。
测试时提示调优能提升视觉语言模型的零样本泛化能力，但通常忽略测试样本间的关联性。在线测试时提示调优虽可利用先前测试样本信息，却存在因误差累积导致提示坍塌的风险。为改进测试时提示调优，我们提出DynaPrompt（动态测试时提示调优），在利用相关数据分布信息的同时减少误差累积。基于在线提示缓冲区，DynaPrompt在调优过程中动态选择并优化每个测试样本的相关提示。具体而言，我们引入基于预测熵和概率差的动态提示选择策略；针对未见测试数据信息，设计了动态提示追加机制，支持缓冲区新增提示并移除无效提示。由此，提示得以在特定测试数据上充分挖掘有益信息，同时缓解误差累积。在14个数据集上的实验验证了动态测试时提示调优的有效性。
原文摘要 · Abstract (English)
Test-time prompt tuning enhances zero-shot generalization of vision-language models but tends to ignore the relatedness among test samples during inference. Online test-time prompt tuning provides a simple way to leverage the information in previous test samples, albeit with the risk of prompt collapse due to error accumulation. To enhance test-time prompt tuning, we propose DynaPrompt, short for dynamic test-time prompt tuning, exploiting relevant data distribution information while reducing error accumulation. Built on an online prompt buffer, DynaPrompt adaptively selects and optimizes the relevant prompts for each test sample during tuning. Specifically, we introduce a dynamic prompt selection strategy based on two metrics: prediction entropy and probability difference. For unseen test data information, we develop dynamic prompt appending, which allows the buffer to append new prompts and delete the inactive ones. By doing so, the prompts are optimized to exploit beneficial information on specific test data, while alleviating error accumulation. Experiments on fourteen datasets demonstrate the effectiveness of dynamic test-time prompt tuning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。