轻量级哈希模型实现在线跨模态检索,效果超越现有方法。
Lightweight Contrastive Distilled Hashing for Online Cross-modal Retrieval
- 用相似矩阵近似实现离线与在线哈希知识迁移。
- 在三个数据集上优于当前最优方法,性能显著提升。
- 适合实时流数据场景的高效跨模态检索应用。
深度在线跨模态哈希因其低存储、高效率和跨模态自适应等优势受到广泛关注。然而仍存在若干技术难题:如何提取跨模态数据的共现语义相关性;如何在处理实时数据流时保持优异性能;如何以轻量方式将离线学习的知识迁移到在线训练。为此,本文提出轻量级对比蒸馏哈希(LCDH),通过知识蒸馏框架中的相似矩阵近似,创新性地连接离线与在线跨模态哈希。教师网络使用对比语言-图像预训练(CLIP)提取跨模态特征,经特征融合后进入注意力模块增强表示,再通过全连接层生成适配相似矩阵尺寸的哈希码。学生网络采用轻量模型提取视觉与文本特征,经全连接层输出二进制码。通过近似相似矩阵,学生网络可获得来自教师网络的共现语义监督,从而提升在线哈希性能。在三个主流数据集上的实验表明,LCDH优于多个前沿方法。
原文摘要 · Abstract (English)
Deep online cross-modal hashing has gained much attention from researchers recently, as its promising applications with low storage requirement, fast retrieval efficiency and cross modality adaptive, etc. However, there still exists some technical hurdles that hinder its applications, e.g., 1) how to extract the coexistent semantic relevance of cross-modal data, 2) how to achieve competitive performance when handling the real time data streams, 3) how to transfer the knowledge learned from offline to online training in a lightweight manner. To address these problems, this paper proposes a lightweight contrastive distilled hashing (LCDH) for cross-modal retrieval, by innovatively bridging the offline and online cross-modal hashing by similarity matrix approximation in a knowledge distillation framework. Specifically, in the teacher network, LCDH first extracts the cross-modal features by the contrastive language-image pre-training (CLIP), which are further fed into an attention module for representation enhancement after feature fusion. Then, the output of the attention module is fed into a FC layer to obtain hash codes for aligning the sizes of similarity matrices for online and offline training. In the student network, LCDH extracts the visual and textual features by lightweight models, and then the features are fed into a FC layer to generate binary codes. Finally, by approximating the similarity matrices, the performance of online hashing in the lightweight student network can be enhanced by the supervision of coexistent semantic relevance that is distilled from the teacher network. Experimental results on three widely used datasets demonstrate that LCDH outperforms some state-of-the-art methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。