arXiv:2511.19920cs.CV2025-11

用检测模型+视觉大模型融合,实现精准的细粒度图像搜索。

Intelligent Image Search Algorithms Fusing Visual Large Models

  • 先用YOLO快速筛查物体组件,再用视觉大模型验证并判断状态。
  • 在车辆部件数据集上达到94.82%的检索准确率,零样本搜索达94.95%。
  • 适合需要识别新物体或状态(如戴口罩)的工业质检与安防场景。

细粒度图像检索旨在查找包含特定物体组件并评估其详细状态的图像,在安全和工业检测领域至关重要。然而传统方法存在显著局限:手工特征(如SIFT)鲁棒性差;基于深度学习的检测器(如YOLO)虽可识别组件存在,但无法进行状态特定检索或零样本搜索;视觉大模型(VLMs)具备语义理解与零样本能力,但空间定位不准且计算成本高,难以直接用于检索。为此,本文提出DetVLM框架,创新性地将目标检测与视觉大模型融合。该框架采用两阶段流程:首先由YOLO检测器高效、高召回率地筛查组件存在;随后由VLM作为召回增强单元,对漏检组件进行二次验证。该架构直接支持两项先进功能:1)状态搜索——通过任务提示引导VLM验证组件存在并执行复杂状态判断(如“遮阳板降下”),实现基于状态的检索;2)零样本搜索——利用VLM固有的零样本能力,无需特定训练即可识别并检索包含未见组件或属性(如“司机戴口罩”)的图像。在车辆组件数据集上的实验表明,DetVLM达到94.82%的最高总体检索准确率,显著优于仅依赖检测的基线。其在司机戴口罩的零样本搜索中准确率达94.95%,状态搜索任务平均准确率超90%。

原文摘要 · Abstract (English)

Fine-grained image retrieval, which aims to find images containing specific object components and assess their detailed states, is critical in fields like security and industrial inspection. However, conventional methods face significant limitations: manual features (e.g., SIFT) lack robustness; deep learning-based detectors (e.g., YOLO) can identify component presence but cannot perform state-specific retrieval or zero-shot search; Visual Large Models (VLMs) offer semantic and zero-shot capabilities but suffer from poor spatial grounding and high computational cost, making them inefficient for direct retrieval. To bridge these gaps, this paper proposes DetVLM, a novel intelligent image search framework that synergistically fuses object detection with VLMs. The framework pioneers a search-enhancement paradigm via a two-stage pipeline: a YOLO detector first conducts efficient, high-recall component-level screening to determine component presence; then, a VLM acts as a recall-enhancement unit, performing secondary verification for components missed by the detector. This architecture directly enables two advanced capabilities: 1) State Search: Guided by task-specific prompts, the VLM refines results by verifying component existence and executing sophisticated state judgments (e.g., "sun visor lowered"), allowing retrieval based on component state. 2) Zero-shot Search: The framework leverages the VLM's inherent zero-shot capability to recognize and retrieve images containing unseen components or attributes (e.g., "driver wearing a mask") without any task-specific training. Experiments on a vehicle component dataset show DetVLM achieves a state-of-the-art overall retrieval accuracy of 94.82\%, significantly outperforming detection-only baselines. It also attains 94.95\% accuracy in zero-shot search for driver mask-wearing and over 90\% average accuracy in state search tasks.

图像检索视觉大模型细粒度零样本

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。