通过语义一致的提示学习,提升零样本草图图像检索的泛化能力。
SeCo-SBIR: Semantically Consistent Prompt Learning for Zero-Shot Sketch-Based Image Retrieval

- 用文本引导的多模态提示将语义知识注入视觉路径,增强跨域适应性。
- 在三个标准基准上均达最优,类别、泛化和跨数据集设置下表现领先。
- 适合需要强泛化能力的零样本图像检索场景,尤其关注避免过拟合。
通过提示学习将CLIP适配于零样本草图图像检索(ZS-SBIR)面临根本矛盾:模型需通过任务特定调整弥合草图-照片域差距,但额外灵活性可能导致对已见类别的过拟合,削弱CLIP的零样本泛化能力。本文提出SeCo-SBIR,一种语义一致的提示学习框架,从两方面解决此矛盾。首先,采用文本引导的多模态提示策略,将可学习提示向量经由CLIP文本编码器,并通过可学习耦合函数在每一层投影至视觉编码器。由于文本编码器已从大规模语言监督中学习到稳健的抽象类别语义,该机制直接将可迁移的语义知识注入视觉路径,在适应草图-照片域的同时,天然偏向未见类别的泛化。其次,基于扰动的一致性约束通过非对称InfoNCE目标,利用冻结的CLIP参考分支与可训练分支对齐,缓解可学习耦合函数带来的残余过拟合风险——扰动输入输入冻结分支,干净输入输入可训练分支,锚定学习表征于CLIP的可泛化特征空间。结合轻量级适配器与包含三元组、NT-Xent和分类项的多目标损失,SeCo-SBIR在所有三个标准ZS-SBIR基准上,涵盖类别、广义及跨数据集设置,均取得最先进结果。
原文摘要 · Abstract (English)
Adapting CLIP for zero-shot sketch-based image retrieval (ZS-SBIR) via prompt learning faces a fundamental tension: the model must bridge the sketch-photo domain gap through task-specific adaptation, yet the added flexibility risks overfitting to seen training categories and eroding CLIP's zero-shot generalization. We present SeCo-SBIR, a semantically consistent prompt learning framework that resolves this tension from both sides. First, a text-guided multi-modal prompting strategy routes learnable prompt vectors through CLIP's text encoder and projects the resulting intermediate representations into the visual encoder at every layer via learnable coupling functions. Because the text encoder has already learned robust, abstract category-level semantics from large-scale language supervision, this mechanism injects transferable semantic knowledge directly into the visual pathway - adapting the model to the sketch-photo domain while inherently favoring generalization to unseen classes. Second, a perturbation-based consistency constraint addresses the residual overfitting risk from the learnable coupling functions by aligning the adapted model with a frozen CLIP reference branch using an asymmetric InfoNCE objective - augmented inputs feed the frozen branch while clean inputs feed the trainable branch - anchoring the learned representations to CLIP's generalizable feature space. Together with lightweight adapters and a multi-objective loss combining triplet, NT-Xent, and classification terms, SeCo-SBIR achieves state-of-the-art results on all three standard ZS-SBIR benchmarks across categorical, generalized, and across-dataset settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。