用大模型自动生成物体名称，实现无需标签的零样本目标检测
LLM-Guided Agentic Object Detection for Open-World Understanding
- 通过大模型生成场景特定的物体名称，驱动开放词汇检测器定位新物体
- 在LVIS、COCO和COCO-OOD上实现对未知物体的精准定位与命名
- 提出两个新评估指标，分别衡量定位与命名能力，适合开放世界应用
传统目标检测依赖固定类别集，新增物体需耗时重训练。尽管开放世界与开放词汇检测（OWOD 和 OVOD）提升了灵活性，但 OWOD 缺乏未知物体的语义标签，而 OVOD 依赖用户提示，限制自主性。我们提出一种基于大语言模型（LLM）的智能体目标检测框架（LAOD），实现完全无标签、零样本检测：通过提示 LLM 生成场景相关的物体名称，传入开放词汇检测器进行定位，使系统能动态调整检测目标。我们引入两个新指标——类无关平均精度（CAAP）和语义命名平均精度（SNAP），分别评估定位与命名性能。在 LVIS、COCO 及 COCO-OOD 数据集上的实验验证了该方法在检测与命名新物体上的优异表现，显著提升开放世界理解中的自主性与适应性。
原文摘要 · Abstract (English)
Object detection traditionally relies on fixed category sets, requiring costly re-training to handle novel objects. While Open-World and Open-Vocabulary Object Detection (OWOD and OVOD) improve flexibility, OWOD lacks semantic labels for unknowns, and OVOD depends on user prompts, limiting autonomy. We propose an LLM-guided agentic object detection (LAOD) framework that enables fully label-free, zero-shot detection by prompting a Large Language Model (LLM) to generate scene-specific object names. These are passed to an open-vocabulary detector for localization, allowing the system to adapt its goals dynamically. We introduce two new metrics, Class-Agnostic Average Precision (CAAP) and Semantic Naming Average Precision (SNAP), to separately evaluate localization and naming. Experiments on LVIS, COCO, and COCO-OOD validate our approach, showing strong performance in detecting and naming novel objects. Our method offers enhanced autonomy and adaptability for open-world understanding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。