用语言模型让短查询生成高质量图像检索结果
Seeing Through Words: Controlling Visual Retrieval Quality with Language Models
- 用生成式语言模型扩展短查询,加入姿态、场景等细节
- 通过质量分级控制,使检索结果更符合用户偏好
- 无需修改现有模型,适合普通用户快速提升检索效果
文本到图像的检索是视觉-语言学习中的基础任务,但在真实场景中常因用户查询过短且描述不清而受阻。这类查询通常仅一两个词,语义模糊,易引发视觉歧义,且无法控制检索图像的质量。为此,我们提出一种可调控质量的检索新范式:利用生成式语言模型作为查询补全函数,将简略查询扩展为包含姿态、场景、美学等细粒度视觉属性的描述性表达,并基于相关性与美学评分模型定义离散的质量等级,使查询补全兼具语义意义和质量感知。该系统具有三大优势:1)灵活性,兼容任意预训练视觉-语言模型(VLMs)无需修改;2)透明性,补全后的查询对用户可解释;3)可控性,支持按用户偏好引导检索结果质量。大量实验表明,本方法显著提升检索性能并实现有效质量控制,弥合了现代VLM表达能力与短查询描述不足之间的差距。代码已开源:https://github.com/Jianglin954/QCQC。
原文摘要 · Abstract (English)
Text-to-image retrieval is a fundamental task in vision-language learning, yet in real-world scenarios it is often challenged by short and underspecified user queries. Such queries are typically only one or two words long, rendering them semantically ambiguous, prone to collisions across diverse visual interpretations, and lacking explicit control over the quality of retrieved images. To address these issues, we propose a new paradigm of quality-controllable retrieval, which enriches short queries with contextual details while incorporating explicit notions of image quality. Our key idea is to leverage a generative language model as a query completion function, extending underspecified queries into descriptive forms that capture fine-grained visual attributes such as pose, scene, and aesthetics. We introduce a general framework that conditions query completion on discretized quality levels, derived from relevance and aesthetic scoring models, so that query enrichment is not only semantically meaningful but also quality-aware. The resulting system provides three key advantages: 1) flexibility, it is compatible with any pretrained vision-language model (VLMs) without modification; 2) transparency, enriched queries are explicitly interpretable by users; and 3) controllability, enabling retrieval results to be steered toward user-preferred quality levels. Extensive experiments demonstrate that our proposed approach significantly improves retrieval results and provides effective quality control, bridging the gap between the expressive capacity of modern VLMs and the underspecified nature of short user queries. Our code is available at https://github.com/Jianglin954/QCQC.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。