解决视觉模型对图像分块位置敏感的问题，提升跨场景泛化能力
PPOM: Marginalizing Patch-Grid Phase for CLIP-Based Generalizable Vision-Language Prompt Tuning
- 引入无训练推理算子PPOM，将分块相位视为干扰变量
- 通过镜像平移与反向配对，均衡不同分块位置的预测结果
- 无需重训练即可提升多种提示学习模型的性能
提示微调虽能以极少参数适配基于CLIP的视觉语言模型，但其预测仍受冻结视觉变压器的空间采样方式影响。特别是非重叠分块令牌化使预测依赖于图像与分块网格的对齐（相位）。为降低对分块网格对齐的敏感性，我们提出分块相位轨道消融（PPOM），一种无训练的推理算子，将相位偏移视为干扰变量。给定分块步长，PPOM评估原视图和镜像填充平移视图，将相反位移配对为水平、垂直和对角反向家族，并对这些家族及原预测赋予相等权重，避免相位融合中的视图计数偏差。总之，PPOM在提示适应与分块网格敏感性之间提供了确定性接口。在多个提示学习主干上，PPOM均在不重新训练的情况下提升了主干性能。
原文摘要 · Abstract (English)
Prompt tuning adapts CLIP-based vision-language models with few trainable parameters, yet its predictions remain sensitive to the spatial sampling imposed by a frozen vision transformer. In particular, non-overlapping patch tokenization makes predictions depend on the alignment (phase) between image and the patch lattice. To reduce prediction sensitivity to patch-grid alignment, we introduce Patch-Phase Orbit Marginalization (PPOM), a training-free inference operator that treats phase shift as a nuisance variable. Given a patch stride, PPOM evaluates the identity view and reflection-padded translations, pairs opposite shifts into horizontal, vertical, and diagonal antithetic families, and assigns equal mass to these families and the identity prediction to avoid view-count bias during phase integration. In summary, PPOM provides a deterministic interface between prompt adaptation and patch-grid sensitivity. Across multiple prompt-learning hosts, PPOM improves host performance without re-training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。