通过筛选关键视觉特征，提升提示调优的准确率与速度
Spotlighter: Revisiting Prompt Tuning from a Representative Mining View
- 基于样本和语义双重评估，筛选最具代表性的视觉标记
- 在11个少样本任务中准确率最高提升11.19%，速度提升0.8K FPS
- 仅增加21个参数，适合资源受限场景下的高效模型优化
CLIP的成功表明提示调优可在开放域识别到细粒度分类等任务中实现稳健的跨模态语义对齐。然而，冗余或弱相关特征会引入噪声并增加计算开销。本文提出Spotlighter，一种轻量级标记选择框架，同时提升提示调优的精度与效率。Spotlighter从样本层面和语义层面评估每个视觉标记的激活程度，仅保留得分最高的标记用于下游预测。一个类特定的语义记忆库通过学习原型优化选择过程，确保语义代表性并补偿被丢弃的特征。为进一步强调信息量高的信号，引入两级排序机制，动态加权标记与原型间的交互。在11个少样本基准测试中，Spotlighter相较于CLIP最高提升11.19%的调和平均准确率，且速度提升0.8K FPS，仅增加21个额外参数。这些结果确立了Spotlighter作为提示调优的有效且可扩展的基线方法。代码将发布于https://github.com/greatest-gourmet/Spotlighter。
原文摘要 · Abstract (English)
CLIP's success has demonstrated that prompt tuning can achieve robust cross-modal semantic alignment for tasks ranging from open-domain recognition to fine-grained classification. However, redundant or weakly relevant feature components introduce noise and incur unnecessary computational costs. In this work, we propose Spotlighter, a lightweight token-selection framework that simultaneously enhances accuracy and efficiency in prompt tuning. Spotlighter evaluates each visual token's activation from both sample-wise and semantic-wise perspectives and retains only the top-scoring tokens for downstream prediction. A class-specific semantic memory bank of learned prototypes refines this selection, ensuring semantic representativeness and compensating for discarded features. To further prioritize informative signals, we introduce a two-level ranking mechanism that dynamically weights token--prototype interactions. Across 11 few-shot benchmarks, Spotlighter outperforms CLIP by up to 11.19\% in harmonic mean accuracy and achieves up to 0.8K additional FPS, with only 21 extra parameters. These results establish Spotlighter as an effective and scalable baseline for prompt tuning. Code for our method will be available at https://github.com/greatest-gourmet/Spotlighter.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。