arXiv:2501.15379cs.IRcs.AI2025-01被引 7

不用微调大模型,用扩散模型提升图文检索的泛化能力。

Diffusion Augmented Retrieval: A Training-Free Approach to Interactive Text-to-Image Retrieval

  • 用对话和扩散模型生成多视角图像表征,增强查询理解。
  • 复杂查询下,十轮对话后命中率比微调模型高7.61%。
  • 适合需要快速响应、应对多样查询的电商与教育场景。

交互式文本到图像检索(I-TIR)是电商、教育等领域前沿服务的重要支撑。然而现有方法依赖微调的多模态大模型(MLLM),训练成本高且泛化能力差。原因在于:1)微调会缩小预训练分布,限制泛化性;2)交互式查询日益多样化和复杂化,导致模型难以覆盖未见样本。为此,我们提出扩散增强检索(DAR),利用扩散模型(DMs)进行文本到图像映射,避免微调大模型,同时保持对复杂查询的鲁棒性能。DAR通过大语言模型对话优化生成多个中间表示,并结合扩散模型生成更丰富的语义-视觉表征,从而更准确地匹配相关图像。在四个基准上的实验表明:简单查询下,DAR性能与微调模型相当,却无需训练开销;当查询经十轮对话变复杂时,其Hits@10较微调模型最高提升7.61%,展现出更强的泛化能力。

原文摘要 · Abstract (English)

Interactive Text-to-image retrieval (I-TIR) is an important enabler for a wide range of state-of-the-art services in domains such as e-commerce and education. However, current methods rely on finetuned Multimodal Large Language Models (MLLMs), which are costly to train and update, and exhibit poor generalizability. This latter issue is of particular concern, as: 1) finetuning narrows the pretrained distribution of MLLMs, thereby reducing generalizability; and 2) I-TIR introduces increasing query diversity and complexity. As a result, I-TIR solutions are highly likely to encounter queries and images not well represented in any training dataset. To address this, we propose leveraging Diffusion Models (DMs) for text-to-image mapping, to avoid finetuning MLLMs while preserving robust performance on complex queries. Specifically, we introduce Diffusion Augmented Retrieval (DAR), a framework that generates multiple intermediate representations via LLM-based dialogue refinements and DMs, producing a richer depiction of the user's information needs. This augmented representation facilitates more accurate identification of semantically and visually related images. Extensive experiments on four benchmarks show that for simple queries, DAR achieves results on par with finetuned I-TIR models, yet without incurring their tuning overhead. Moreover, as queries become more complex through additional conversational turns, DAR surpasses finetuned I-TIR models by up to 7.61% in Hits@10 after ten turns, illustrating its improved generalization for more intricate queries.

图文检索扩散模型零样本

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。