不微调模型,用内在视觉证据提升图像描述准确性与完整性。
PatchGate: Narrowing the Verbalization Gap with Intrinsic Object Inventories in Frozen Vision-Language Models

- 从冻结模型中提取图像级对象证据,无需额外提示
- 提升可见物体覆盖率至56.0%,降低幻觉物体数12.0%
- 适合追求高可靠性的零样本图像描述场景
可靠的图像描述需兼顾准确性和完整性,避免提及未见物体的同时覆盖所有可见物体。现有免训练方法主要抑制无关提及,但难以恢复被遗漏的可见物体。本文提出PatchGate,一种免训练框架,在生成前提取冻结视觉语言模型中固有的补丁级语义证据,并用于缩小内在对象集与最终提及之间的差距。第一阶段,视觉证据提取(VEX)读取语言模型解码器后半部分的补丁级词汇证据,构建无任务提示的图像条件对象集;第二阶段,视觉证据包含排除解码(VIED)利用该证据校准解码逻辑,促进被支持但未充分描述的对象,抑制弱支持却过度提及的对象。在AMBER数据集上,PatchGate将可见物体覆盖率从49.4%提升至56.0%(+13.4%),同时将CHAIR值从7.5降至6.6(-12.0%),无需外部检测器或微调,仅增加一次前向传播。
原文摘要 · Abstract (English)
Reliable image captioning in Vision-Language Models (VLMs) requires captions to be both precise and complete, avoiding unsupported object mentions while covering visible objects. Existing training-free methods primarily address the former requirement, suppressing unsupported object words by intervening on model-predicted mentions during generation. Because they operate only on objects the model is already likely to mention, visible objects omitted from the output remain difficult to recover. We propose PatchGate, a training-free framework that extracts prompt-free object evidence intrinsic to a frozen VLM before generation and uses it to narrow the gap between an intrinsic object set and final object mentions. In the first stage, Visual Evidence eXtraction (VEX) reads patch-level lexical evidence from the latter half of LM decoder layers and constructs an image-conditioned object set without any task prompt. In the second stage, Visual-Evidence Inclusion-Exclusion Decoding (VIED) uses this object evidence to calibrate decoding logits, promoting evidence-supported but under-verbalized objects and suppressing weakly supported but over-verbalized objects. On AMBER, PatchGate improves both sides of object-level reliability, increasing visible-object coverage from 49.4 to 56.0 (+13.4%) and reducing object hallucination by lowering CHAIR from 7.5 to 6.6 (-12.0%), without external detectors or fine-tuning and with one extra forward pass.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。