arXiv:2602.13402cs.HCcs.IR2026-02中稿 · publication in IEE…

InfoCIR让图像+文本搜索结果可解释,直观看提示词如何影响检索。

InfoCIR: Multimedia Analysis for Composed Image Retrieval

  • 通过交互式界面整合检索、解释与提示优化,支持图像+文本联合查询
  • 用UMAP降维可视化结果分布,结合注意力图和关键词归因分析变化原因
  • 适合模型开发者调试提示词,也帮助用户理解为何微小措辞会大幅改变结果

组合图像检索(CIR)允许用户通过参考图像加文本提示来搜索所需修改的图像。尽管视觉语言模型如CLIP已将多模态信息嵌入统一空间,但开发者仍缺乏工具揭示这些多模态提示如何与嵌入空间互动,以及为何微小措辞变化会显著影响结果。我们提出InfoCIR,一个视觉分析系统,通过在一个交互式仪表板中融合检索、可解释性和提示工程,填补这一空白。InfoCIR集成最先进的CIR后端(SEARLE arXiv:2303.15247),配备六面板界面:(i) 支持图像+文本查询组合;(ii) 使用均匀流形近似与投影(UMAP)将前k个结果投影至低维空间以支持空间推理;(iii) 叠加基于相似性的显著性图和梯度衍生的词元归因条,实现局部解释;(iv) 采用基于大语言模型的提示增强器,生成反事实变体,并可视化这些变化如何影响用户选定目标图像的排名。模块化架构基于Plotly-Dash,可轻松接入新模型、数据集和归因方法。我们主张InfoCIR有助于诊断检索失败、指导提示优化,并加速模型开发中的洞察生成。所有源代码及可复现演示均开源于https://github.com/giannhskp/InfoCIR。

原文摘要 · Abstract (English)

Composed Image Retrieval (CIR) allows users to search for images by combining a reference image with a text prompt that describes desired modifications. While vision-language models like CLIP have popularized this task by embedding multiple modalities into a joint space, developers still lack tools that reveal how these multimodal prompts interact with embedding spaces and why small wording changes can dramatically alter the results. We present InfoCIR, a visual analytics system that closes this gap by coupling retrieval, explainability, and prompt engineering in a single, interactive dashboard. InfoCIR integrates a state-of-the-art CIR back-end (SEARLE arXiv:2303.15247) with a six-panel interface that (i) lets users compose image + text queries, (ii) projects the top-k results into a low-dimensional space using Uniform Manifold Approximation and Projection (UMAP) for spatial reasoning, (iii) overlays similarity-based saliency maps and gradient-derived token-attribution bars for local explanation, and (iv) employs an LLM-powered prompt enhancer that generates counterfactual variants and visualizes how these changes affect the ranking of user-selected target images. A modular architecture built on Plotly-Dash allows new models, datasets, and attribution methods to be plugged in with minimal effort. We argue that InfoCIR helps diagnose retrieval failures, guides prompt enhancement, and accelerates insight generation during model development. All source code allowing for a reproducible demo is available at https://github.com/giannhskp/InfoCIR.

图像检索可解释性提示工程

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。