提出关系感知元学习框架,提升零样本手绘图像检索泛化能力
Relation-Aware Meta-Learning for Zero-shot Sketch-Based Image Retrieval
- 设计成对关系感知四元组损失,优化跨模态特征对齐
- 在扩展Sketchy和TU-Berlin数据集上,零样本检索准确率显著超越现有方法
- 适合需要跨模态泛化能力的研究者,尤其关注零样本图像检索场景
手绘图像检索(SBIR)依赖自由绘制的草图在同类别中检索自然照片。然而,其实际应用受限于无法检索训练集中未出现的类别。为此,该任务演变为零样本手绘图像检索(ZS-SBIR),模型需在未见类别上评估性能。传统SBIR主要关注缩小照片与草图之间的领域差距,但在零样本设置下,模型还需具备强泛化能力以将知识迁移至未见类别。为此,本文提出一种新型框架,采用成对关系感知四元组损失来弥合特征差距。通过引入来自不同模态的两个负样本,该方法防止正样本特征在某一模态上过远而在另一模态上过近,从而增强类间可分性。同时,提出关系感知元学习网络(RAMLN)来动态获取跨模态四元组损失的边距超参数,以提升模型泛化能力。RAMLN利用外部记忆存储特征信息,并据此分配最优边距值。在扩展Sketchy和TU-Berlin数据集上的实验结果表明,该方法在ZS-SBIR任务上显著优于现有最先进方法。
原文摘要 · Abstract (English)
Sketch-based image retrieval (SBIR) relies on free-hand sketches to retrieve natural photos within the same class. However, its practical application is limited by its inability to retrieve classes absent from the training set. To address this limitation, the task has evolved into Zero-Shot Sketch-Based Image Retrieval (ZS-SBIR), where model performance is evaluated on unseen categories. Traditional SBIR primarily focuses on narrowing the domain gap between photo and sketch modalities. However, in the zero-shot setting, the model not only needs to address this cross-modal discrepancy but also requires a strong generalization capability to transfer knowledge to unseen categories. To this end, we propose a novel framework for ZS-SBIR that employs a pair-based relation-aware quadruplet loss to bridge feature gaps. By incorporating two negative samples from different modalities, the approach prevents positive features from becoming disproportionately distant from one modality while remaining close to another, thus enhancing inter-class separability. We also propose a Relation-Aware Meta-Learning Network (RAMLN) to obtain the margin, a hyper-parameter of cross-modal quadruplet loss, to improve the generalization ability of the model. RAMLN leverages external memory to store feature information, which it utilizes to assign optimal margin values. Experimental results obtained on the extended Sketchy and TU-Berlin datasets show a sharp improvement over existing state-of-the-art methods in ZS-SBIR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。