无需微调，用智能推理实现精准图文定位
Connecting the Dots: Training-Free Visual Grounding via Agentic Reasoning
- 构建迭代式推理框架，融合检测器与大模型逐步优化定位区域
- 零样本准确率达65.1%，仅用查询文本就达90%选择准确率
- 全程可解释，适合需要透明决策的视觉语言应用
视觉定位任务旨在将文本查询与图像中的特定区域关联，是视觉-语言融合的关键。现有方法依赖大量特定任务标注与微调，难以泛化到新场景。为此，我们提出GroundingAgent，一种无需任务微调的智能视觉定位框架。该框架通过结构化、迭代式推理，结合预训练开放词汇目标检测器、多模态大语言模型（MLLM）和大语言模型（LLM），实现语义与空间分析的联合优化。令人瞩目的是，GroundingAgent在主流基准（RefCOCO、RefCOCO+、RefCOCOg）上达到65.1%的平均零样本定位准确率。进一步地，仅使用原始查询文本替代MLLM生成描述时，选择阶段准确率接近90%，接近监督性能，凸显了LLM推理能力的核心作用。该方法还具备强可解释性，清晰展示每一步推理过程，揭示决策逻辑。
原文摘要 · Abstract (English)
Visual grounding, the task of linking textual queries to specific regions within images, plays a pivotal role in vision-language integration. Existing methods typically rely on extensive task-specific annotations and fine-tuning, limiting their ability to generalize effectively to novel or out-of-distribution scenarios. To address these limitations, we introduce GroundingAgent, a novel agentic visual grounding framework that operates without any task-specific fine-tuning. GroundingAgent employs a structured, iterative reasoning mechanism that integrates pretrained open-vocabulary object detectors, multimodal large language models (MLLMs), and large language models (LLMs) to progressively refine candidate regions through joint semantic and spatial analyses. Remarkably, GroundingAgent achieves an average zero-shot grounding accuracy of 65.1 % on widely-used benchmarks (RefCOCO, RefCOCO+, RefCOCOg), entirely without fine-tuning. Furthermore, by substituting MLLM-generated captions with the original query texts, the accuracy at the selection stage alone reaches approximately 90 %, closely matching supervised performance and underscoring the critical role of LLM reasoning capabilities. GroundingAgent also offers strong interpretability, transparently illustrating each reasoning step and providing clear insights into its decision-making process.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。