提出U形多粒度提示框架，提升视觉语言模型的细粒度感知与泛化能力。
U-shaped Multi-granularity Learning for Vision-Language Models
- 借鉴U-Net结构，构建视觉与文本双模态并行多粒度表示
- 在MSCOCO上比MAMET和VPKE分别提升4.1和7.3 rSum
- 适合需要跨任务泛化与高效推理的视觉语言模型研究者
视觉语言模型的提示学习虽有效，却面临粒度困境：全局提示缺乏细粒度语义感知，局部提示忽略上下文关联，限制跨任务泛化能力。该问题在密集预测任务中尤为显著。受U-Net启发，我们提出UPrompt，一种用于视觉语言模型的U形多粒度提示学习框架。类似U-Net通过对称编码器-解码器路径与跨层连接融合多层级特征，UPrompt在视觉与文本模态中并行构建多粒度表征，粗粒度到细粒度的级联增强将全局上下文传递以细化局部细节，而细粒度到粗粒度的层级监督确保各尺度间语义一致性。在17个基准测试上的大量实验验证了其有效性。UPrompt在MSCOCO上比MAMET和VPKE分别提升4.1和7.3 rSum，base-to-novel泛化能力超越CoCoA-Mix达5.09%，同时保持极低开销（粗粒度）且中等粒度成本仅为PSRC的1/3。
原文摘要 · Abstract (English)
The prompt learning paradigm for vision-language models is effective yet faces a granularity dilemma: global prompts lack fine-grained semantic awareness, while local prompts ignore contextual associations, limiting cross-task generalization. This dilemma exists in dense prediction tasks. Inspired by U-Net, which unifies multi-level representations across granularities, we propose UPrompt, a U-shaped multi-granularity prompt learning framework for vision-language models. Similar to how U-Net integrates fine and coarse features through symmetric encoder-decoder pathways with cross-level connections, UPrompt constructs parallel multi-granularity representations in both visual and textual modalities, where coarse-to-fine cascaded enhancement propagates global context to refine local details, while fine-to-coarse hierarchical supervision ensures semantic consistency across scales. Extensive experiments on 17 benchmarks validate our effectiveness. UPrompt outperforms MAMET and VPKE by 4.1 and 7.3 rSum on MSCOCO, surpasses CoCoA-Mix by 5.09% in base-to-novel generalization, while maintaining competitive performance with minimal overhead (coarse-grained) and matching PSRC with 1/3 cost (medium-grained).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。