arXiv:2606.28862cs.CV2026-06

提出轻量绑定钩子,让冻结检测器更准地对齐语言查询与图像区域。

HKVLM: Faithful Query--Region Binding for Frozen-Detector Visual Grounding

论文配图:HKVLM: Faithful Query--Region Binding for Frozen-Detector Visual Grounding
图 1 · 摘自论文原文
  • 用轻量钩子将语言查询映射到检测框,实现精准对齐。
  • 通过拒答机制减少幻觉,错误率下降30%以上。
  • 适合研究冻结模型下视觉定位的绑定问题。

视觉定位常因语言指代与错误区域绑定而失败,即使目标物体在候选区域中存在。本文在冻结感知条件下研究这一绑定失败问题,探索是否可通过显式查询-区域对齐钩子和基于感知的拒答机制,在不重训练检测器或视觉语言主干的情况下提升定位忠实性。HKVLM冻结一个语言对齐的开放词汇检测器,学习一个轻量级钩子,将查询嵌入映射到检测器提议框的共享空间;同时引入验证器,当无区域充分支持查询时选择拒答。本文证明了精确的提案级诊断分解公式 $(1- ext{SeeErr})(1- ext{SayErr})$,分离提案覆盖失败与条件绑定失败,并揭示拒答带来的忠实性-召回率权衡的单调性。在 RefCOCO、RefCOCO+、RefCOCOg 和 POPE 数据集上,HKVLM 超过未训练和训练过的匹配感知绑定对照组,显著降低幻觉。尽管坐标解码和端到端微调基线在原始准确率上仍更高,但推理压力测试暴露绑定仍是当前主要瓶颈。因此,本文将 HKVLM 视为冻结感知下查询-区域绑定的诊断与机制研究,而非绝对定位领导者。

原文摘要 · Abstract (English)

Visual grounding often fails even when the target object is present in the proposal pool, because the language-side referent is bound to the wrong region. We study this binding failure under frozen perception and ask whether an explicit query--region alignment hook, together with a perception-grounded abstention mechanism, can improve faithful grounding without retraining the detector or the vision-language backbone. HKVLM freezes a language-aligned open-vocabulary detector for localization and learns a lightweight hook that maps referential query embeddings to detector proposals in a shared space; a verifier abstains when no region sufficiently supports the query. We prove an exact proposal-level diagnostic decomposition, $(1-\mathrm{SeeErr})(1-\mathrm{SayErr})$, separating proposal-coverage failures from conditional binding failures, and a monotonicity result that characterizes the faithfulness--recall trade-off induced by abstention. Across RefCOCO, RefCOCO+, RefCOCOg, and POPE, HKVLM improves over untrained and trained matched-perception binding controls and substantially reduces hallucination through abstention. Strong coordinate-decoding and end-to-end fine-tuned baselines remain much higher in raw grounding accuracy, and a reasoning-stress set exposes binding as the main current bottleneck. We therefore present HKVLM as a diagnostic and mechanism-level study of query--region binding under frozen perception, not as an absolute localization leader.

视觉定位冻结模型绑定机制拒答

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。