用大模型增强查询与重排,实现无需训练的精准图像组合检索
SQUARE: Semantic Query-Augmented Fusion and Efficient Batch Reranking for Training-free Zero-Shot Composed Image Retrieval
- 通过大模型生成目标图像描述,丰富查询语义
- 单次遍历完成批量重排,显著提升检索精度
- 轻量模型也表现优异,适合实际部署
组合图像检索(CIR)旨在找到既保留参考图像视觉内容,又融入用户指定文本修改的目标图像。无需任务特定训练或标注数据的免训练零样本CIR(ZS-CIR)极具吸引力,但准确捕捉用户意图仍具挑战。本文提出SQUARE,一种两阶段免训练框架,利用多模态大语言模型(MLLM)增强ZS-CIR。在语义查询增强融合(SQAF)阶段,将基于视觉-语言模型(如CLIP)的查询嵌入,与由MLLM生成的目标图像描述结合,提供高层语义引导,更好捕捉用户意图,提升全局检索质量。在高效批量重排(EBR)阶段,将排名靠前的候选图像以带视觉标记的网格形式呈现给MLLM,进行跨候选图像的联合视觉-语义推理。该重排策略单次遍历完成,实现更精确排序。实验表明,SQUARE在四个标准CIR基准上表现强劲,且在轻量预训练模型下仍保持高效率,具备广泛应用潜力。
原文摘要 · Abstract (English)
Composed Image Retrieval (CIR) aims to retrieve target images that preserve the visual content of a reference image while incorporating user-specified textual modifications. Training-free zero-shot CIR (ZS-CIR) approaches, which require no task-specific training or labeled data, are highly desirable, yet accurately capturing user intent remains challenging. In this paper, we present SQUARE, a novel two-stage training-free framework that leverages Multimodal Large Language Models (MLLMs) to enhance ZS-CIR. In the Semantic Query-Augmented Fusion (SQAF) stage, we enrich the query embedding derived from a vision-language model (VLM) such as CLIP with MLLM-generated captions of the target image. These captions provide high-level semantic guidance, enabling the query to better capture the user's intent and improve global retrieval quality. In the Efficient Batch Reranking (EBR) stage, top-ranked candidates are presented as an image grid with visual marks to the MLLM, which performs joint visual-semantic reasoning across all candidates. Our reranking strategy operates in a single pass and yields more accurate rankings. Experiments show that SQUARE, with its simplicity and effectiveness, delivers strong performance on four standard CIR benchmarks. Notably, it maintains high performance even with lightweight pre-trained, demonstrating its potential applicability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。