通过稀疏最优传输实现局部全局提示协同，提升少样本分类与分布外检测性能。
Local-Global Prompt Learning via Sparse Optimal Transport
- 用平衡熵正则最优传输分配视觉区域，避免局部提示重叠。
- 16样本下平均准确率达85.1%，分布外检测AUC达94.2%，超越现有方法。
- 无需可学习投影，保持CLIP原始特征空间结构，适合对鲁棒性要求高的场景。
少样本适配视觉语言模型（如CLIP）通常依赖全局图像嵌入匹配文本提示。近期工作引入局部图像-文本对齐以捕捉细粒度视觉线索，但多数方法独立为每个提示选择局部区域，导致特征冗余和提示重叠。本文提出SOT-GLP，通过共享稀疏补丁支持与平衡最优传输分配，显式划分显著视觉区域给类别相关局部提示，同时保留全局对齐。方法学习共享全局提示与类别特定局部提示：全局分支维持标准图像-文本匹配以实现稳健类别级对齐；局部分支利用V-V注意力构建类别条件稀疏补丁集，并通过平衡熵正则最优传输将其对齐至多个类别提示，实现补丁的软划分，防止提示重叠与坍缩。在两个互补目标上评估：(i) 11个标准基准上的少样本分类准确率；(ii) 分布外（OOD）检测。在16样本ViT-B/16设置下，SOT-GLP在11数据集上达到85.1%平均准确率，优于先前提示学习方法。我们发现提示学习存在显著的准确率-鲁棒性权衡：可学习投影虽优化分布内拟合，却改变基础特征空间。实验表明，无投影的局部对齐能保持CLIP流形的原生几何结构，实现94.2% AUC的顶尖OOD检测性能，超越完全适配模型。代码已开源：https://github.com/Deniz2304988/SOT-GLP。
原文摘要 · Abstract (English)
Few-shot adaptation of vision-language models (VLMs) like CLIP typically relies on learning textual prompts matched to global image embeddings. Recent works extend this paradigm by incorporating local image-text alignment to capture fine-grained visual cues, yet these approaches often select local regions independently for each prompt, leading to redundant local feature usage and prompt overlap. We propose SOT-GLP, which introduces a shared sparse patch support and balanced optimal transport allocation to explicitly partition salient visual regions among class-specific local prompts while preserving global alignment. Our method learns shared global prompts and class-specific local prompts. The global branch maintains standard image-text matching for robust category-level alignment. The local branch constructs a class-conditioned sparse patch set using V-V attention and aligns it to multiple class-specific prompts via balanced entropic optimal transport, yielding a soft partition of patches that prevents prompt overlap and collapse. We evaluate our method on two complementary objectives: (i) few-shot classification accuracy on 11 standard benchmarks and (ii) out-of-distribution (OOD) detection. On the standard 11-dataset benchmark with 16-shot ViT-B/16, SOT-GLP achieves 85.1% average accuracy, outperforming prior prompt-learning methods. We identify a distinct accuracy-robustness trade-off in prompt learning: while learnable projections optimize in-distribution fit, they alter the foundational feature space. We demonstrate that a projection-free local alignment preserves the native geometry of the CLIP manifold, yielding state-of-the-art OOD detection performance (94.2% AUC) that surpasses fully adapted models. Implementation available at: https://github.com/Deniz2304988/SOT-GLP
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。