arXiv:2512.11099cs.CV2025-12被引 4

将推理与定位解耦,用模块化设计提升多目标视觉定位速度和准确率。

VGent: Visual Grounding via Modular Design for Disentangling Reasoning and Prediction

  • 用冻结的多模态大模型做推理,检测框作查询进行定位选择。
  • 在多目标基准上F1提升20.6%,召回率和定位精度显著提高。
  • 适合需要快速高精度定位的工业场景,支持灵活升级模块。

现有视觉定位模型要么依赖多模态大语言模型进行自回归解码,速度慢且易产生幻觉;要么通过重对齐语言模型与视觉特征来学习新对象标记,可能损害预训练推理能力。为此,我们提出VGent,一种模块化的编码器-解码器架构,明确解耦高层推理与低层边界框预测。具体而言,一个冻结的多模态大语言模型作为编码器,保持其强大的原始推理能力;解码器则以检测器生成的高质量候选框为查询,通过跨注意力机制在编码器隐藏状态中选择目标框。该设计充分利用目标检测与多模态大模型进展,避免自回归解码缺陷,实现快速推理。此外,系统支持模块化升级:引入(i)QuadThinker,基于强化学习增强编码器多目标推理能力;(ii)掩码感知标签,解决检测与分割歧义;(iii)全局目标识别,提升对所有目标的识别能力,优化增强候选框间的选框性能。在多目标视觉定位基准上的实验表明,VGent达到新最优,相比先前方法F1提升20.6%,在视觉指代挑战下gIoU提升8.2%、cIoU提升5.8%,同时保持恒定快速推理延迟。

原文摘要 · Abstract (English)

Current visual grounding models are either based on a Multimodal Large Language Model (MLLM) that performs auto-regressive decoding, which is slow and risks hallucinations, or on re-aligning an LLM with vision features to learn new special or object tokens for grounding, which may undermine the LLM's pretrained reasoning ability. In contrast, we propose VGent, a modular encoder-decoder architecture that explicitly disentangles high-level reasoning and low-level bounding box prediction. Specifically, a frozen MLLM serves as the encoder to provide untouched powerful reasoning capabilities, while a decoder takes high-quality boxes proposed by detectors as queries and selects target box(es) via cross-attending on encoder's hidden states. This design fully leverages advances in both object detection and MLLM, avoids the pitfalls of auto-regressive decoding, and enables fast inference. Moreover, it supports modular upgrades of both the encoder and decoder to benefit the whole system: we introduce (i) QuadThinker, an RL-based training paradigm for enhancing multi-target reasoning ability of the encoder; (ii) mask-aware label for resolving detection-segmentation ambiguity; and (iii) global target recognition to improve the recognition of all the targets which benefits the selection among augmented proposals. Experiments on multi-target visual grounding benchmarks show that VGent achieves a new state-of-the-art with +20.6% F1 improvement over prior methods, and further boosts gIoU by +8.2% and cIoU by +5.8% under visual reference challenges, while maintaining constant, fast inference latency.

视觉定位多模态模块化推理解耦

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。