arXiv:2503.16064cs.CVcs.AI2025-03CVPR被引 9

用提示词增强跨模态哈希,提升图文检索精度。

PromptHash: Affinity-Prompted Collaborative Cross-Modal Learning for Adaptive Hashing Retrieval

  • 引入文本亲和提示学习,高效保留上下文信息。
  • 在NUS-WIDE上图文检索准确率提升超18%。
  • 适合需要高精度跨模态检索的场景。

跨模态哈希是实现高效数据检索与存储优化的有前景方法,但现有方法在语义保持、上下文完整性与信息冗余方面存在显著局限,制约了检索效果。本文提出PromptHash,一种基于亲和提示协同学习的自适应跨模态哈希框架。核心贡献包括:(i) 文本亲和提示学习机制,在保持参数效率的同时保留上下文信息;(ii) 自适应门控选择融合架构,结合状态空间模型与Transformer网络,实现精准跨模态特征融合;(iii) 提示亲和对齐策略,通过分层对比学习弥合模态异构性。据我们所知,这是首次系统研究亲和提示在协同跨模态自适应哈希中的应用,建立了跨模态语义一致性增强的新范式。在三个基准多标签数据集上的全面评估表明,该方法显著优于现有方法。尤其在NUS-WIDE数据集上,图像到文本与文本到图像检索任务分别取得18.22%和18.65%的性能提升。代码已开源:https://github.com/ShiShuMo/PromptHash。

原文摘要 · Abstract (English)

Cross-modal hashing is a promising approach for efficient data retrieval and storage optimization. However, contemporary methods exhibit significant limitations in semantic preservation, contextual integrity, and information redundancy, which constrains retrieval efficacy. We present PromptHash, an innovative framework leveraging affinity prompt-aware collaborative learning for adaptive cross-modal hashing. We propose an end-to-end framework for affinity-prompted collaborative hashing, with the following fundamental technical contributions: (i) a text affinity prompt learning mechanism that preserves contextual information while maintaining parameter efficiency, (ii) an adaptive gated selection fusion architecture that synthesizes State Space Model with Transformer network for precise cross-modal feature integration, and (iii) a prompt affinity alignment strategy that bridges modal heterogeneity through hierarchical contrastive learning. To the best of our knowledge, this study presents the first investigation into affinity prompt awareness within collaborative cross-modal adaptive hash learning, establishing a paradigm for enhanced semantic consistency across modalities. Through comprehensive evaluation on three benchmark multi-label datasets, PromptHash demonstrates substantial performance improvements over existing approaches. Notably, on the NUS-WIDE dataset, our method achieves significant gains of 18.22% and 18.65% in image-to-text and text-to-image retrieval tasks, respectively. The code is publicly available at https://github.com/ShiShuMo/PromptHash.

跨模态检索哈希学习提示工程

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。