让视觉与语言提示双向互推,提升模型泛化能力。
Hierarchical Cross-modal Prompt Learning for Vision-Language Models
- 构建跨模态双向知识流,分层融合多尺度语义
- 在11个基准上达最优,显著提升下游任务表现
- 轻量级设计适合实际部署,尤其适合多模态理解
预训练视觉语言模型(如CLIP)具备出色泛化能力,但将其适配到下游任务时仍面临挑战。现有提示学习方法存在模态隔离和语义层级衰减两大瓶颈。为此,本文提出HiCroPL框架,通过建立文本与视觉模态间的双向知识流动,实现语义相互优化。早期层中,文本提示通过分层知识映射器向视觉提示注入清晰语义,增强低层视觉表征;后期层中,包含特定任务对象的视觉提示反馈回文本提示,实现深层对齐。关键在于,分层知识映射器融合多尺度表示,确保深层特征保留可迁移的浅层语义,从而提升泛化性。此外,引入轻量级层特定知识代理,实现高效跨模态交互。在四个任务上的大量实验表明,HiCroPL在11个基准上达到当前最佳性能,显著优于现有方法。代码已开源:https://github.com/zzeoZheng/HiCroPL。
原文摘要 · Abstract (English)
Pre-trained Vision-Language Models (VLMs) such as CLIP have shown excellent generalization abilities. However, adapting these large-scale models to downstream tasks while preserving their generalization capabilities remains challenging. Although prompt learning methods have shown promise, they suffer from two fundamental bottlenecks that limit generalization: (a) modality isolation, and (b) hierarchical semantic decay. To address these limitations, we propose HiCroPL, a Hierarchical Cross-modal Prompt Learning framework that establishes bidirectional knowledge flow between text and vision modalities, enabling them to refine their semantics mutually. HiCroPL routes knowledge flows by leveraging the complementary strengths of text and vision. In early layers, text prompts inject relatively clear semantics into visual prompts through a hierarchical knowledge mapper, enhancing the representation of low-level visual semantics. In later layers, visual prompts encoding specific task-relevant objects flow back to refine text prompts, enabling deeper alignment. Crucially, our hierarchical knowledge mapper allows representations at multi-scales to be fused, ensuring that deeper representations retain transferable shallow semantics thereby enhancing generalization. We further introduce a lightweight layer-specific knowledge proxy to enable efficient cross-modal interactions. Extensive evaluations across four tasks demonstrate HiCroPL's superior performance, achieving state-of-the-art results on 11 benchmarks with significant improvements. Code is available at: https://github.com/zzeoZheng/HiCroPL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。