通过分割聚焦区域提升图像检索精度，解决文本引导不充分问题
OFFSET: Segmentation-based Focus Shift Revision for Composed Image Retrieval
- 基于图像分割识别关键区域，分步提取视觉与文本特征
- 文本引导动态调整关注焦点，显著提升复杂查询检索效果
- 适用于需要精确语义控制的图像检索场景，如设计、电商
组合图像检索（CIR）是一种新型检索范式，支持用户以参考图像加修改文本的多模态查询来检索目标图像。尽管现有方法取得进展，但仍面临两大挑战：一是视觉数据中显著区域与噪声区域差异未被考虑，导致查询特征退化；二是文本在图像修改过程中的优先级被忽视，造成视觉关注偏差。为此，本文提出一种基于聚焦映射的特征提取器，包含主导区域分割和双焦点映射两个模块，能识别图像中的关键部分，并引导视觉与文本特征提取，降低噪声干扰。进一步提出文本引导的焦点修订模块，利用文本中的修改意图对参考图像进行自适应焦点调整，增强组合特征中对修改重点的感知。上述模块共同构成分割式焦点迁移网络（OFFSET），在四个基准数据集上进行了全面实验，验证了该方法的优越性。代码与数据已公开于 https://zivchen-ty.github.io/OFFSET.github.io/
原文摘要 · Abstract (English)
Composed Image Retrieval (CIR) represents a novel retrieval paradigm that is capable of expressing users' intricate retrieval requirements flexibly. It enables the user to give a multimodal query, comprising a reference image and a modification text, and subsequently retrieve the target image. Notwithstanding the considerable advances made by prevailing methodologies, CIR remains in its nascent stages due to two limitations: 1) inhomogeneity between dominant and noisy portions in visual data is ignored, leading to query feature degradation, and 2) the priority of textual data in the image modification process is overlooked, which leads to a visual focus bias. To address these two limitations, this work presents a focus mapping-based feature extractor, which consists of two modules: dominant portion segmentation and dual focus mapping. It is designed to identify significant dominant portions in images and guide the extraction of visual and textual data features, thereby reducing the impact of noise interference. Subsequently, we propose a textually guided focus revision module, which can utilize the modification requirements implied in the text to perform adaptive focus revision on the reference image, thereby enhancing the perception of the modification focus on the composed features. The aforementioned modules collectively constitute the segmentatiOn-based Focus shiFt reviSion nETwork (\mbox{OFFSET}), and comprehensive experiments on four benchmark datasets substantiate the superiority of our proposed method. The codes and data are available on https://zivchen-ty.github.io/OFFSET.github.io/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。