用大模型将类别名转为细粒度属性，提升图像检索泛化能力
Language-driven Fine-grained Retrieval
- 通过大模型将类别名转化为属性描述，构建细粒度监督信号
- 利用视觉语言模型对齐属性并聚类，生成跨类别通用属性词表
- 基于属性原型训练模型，显著提升未见类别检索效果
现有细粒度图像检索方法依赖语义稀疏的独热标签进行监督，虽在已见类别上有效，却忽略了类别名称中的丰富语义，限制了跨类别细节可比性的建模，进而影响对未见类别的泛化能力。为此，我们提出LaFG框架，利用大语言模型（LLM）和视觉-语言模型（VLM）将类别名转化为属性级监督信号。将每个类别名视为语义锚点，通过LLM生成详细、属性导向的描述；为减少属性遗漏，采用冻结的VLM将描述投影至视觉对齐空间，聚类形成全局属性词表，并从相关类别中挖掘互补属性。基于该词表，设计全局提示模板选择类别相关属性，聚合生成类别特异的语义原型，用于监督检索模型训练。
原文摘要 · Abstract (English)
Existing fine-grained image retrieval (FGIR) methods learn discriminative embeddings by adopting semantically sparse one-hot labels derived from category names as supervision. While effective on seen classes, such supervision overlooks the rich semantics encoded in category names, hindering the modeling of comparability among cross-category details and, in turn, limiting generalization to unseen categories. To tackle this, we introduce LaFG, a Language-driven framework for Fine-Grained Retrieval that converts class names into attribute-level supervision using large language models (LLMs) and vision-language models (VLMs). Treating each name as a semantic anchor, LaFG prompts an LLM to generate detailed, attribute-oriented descriptions. To mitigate attribute omission in these descriptions, it leverages a frozen VLM to project them into a vision-aligned space, clustering them into a dataset-wide attribute vocabulary while harvesting complementary attributes from related categories. Leveraging this vocabulary, a global prompt template selects category-relevant attributes, which are aggregated into category-specific linguistic prototypes. These prototypes supervise the retrieval model to steer
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。