arXiv:2608.11074cs.CV2026-08

用细粒度问答评估图像描述真实性,提升模型事实核查能力。

CapProbe: Evaluating Detailed Image Captions via Full-Scene Dense Question Answering

论文配图:CapProbe: Evaluating Detailed Image Captions via Full-Scene Dense Question Answering
图 1 · 摘自论文原文
  • 将图像分区域生成多选题,实现视觉事实逐块验证。
  • 平均每图74个问题,覆盖346张图、25650个问答对。
  • 适合评估生成式模型的细节准确性,尤其关注事实一致性。

评估视觉语言模型生成的详细图像描述,需超越表面语义相似性。现有基于参考文本的指标(如CIDEr、SPICE)和大模型评分协议难以验证密集的事实性陈述,而现有问答类方法普遍存在探针密度低、领域覆盖窄或问题与图像区域未对齐的问题。我们提出CapProbe,一个全场景稠密问答基准,将详细描述评估转化为区域对齐的事实核查。每张图像被分解为涵盖前景与背景的粗粒度语义区域;每个保留区域生成横跨10个语义类别的多项选择题,形成密集的事实检查清单。基于37个一级领域和219个二级子领域的双层分类体系,该基准包含346张图像、1868个区域、25650个问题,平均每个图像74个问答对。语言评判者仅凭描述作答;通过设置‘不确定’选项与有效准确率,提供依赖评判者的代理指标,以区分未回答与错误回答问题,同时密度相关指标惩罚冗长但无信息量的描述。该流程成本可控:将开放式评分转为结构化多选阅读,在减少开放评分偏差的同时保持评判者依赖性,并在固定读者下获得相对稳定的模型排名。13个VLM实验显示模型间存在显著覆盖率差距,明确呈现能力-效率权衡,且暴露了稀疏或重叠评估常忽略的失败模式。数据集、标注与评估代码即将发布。

原文摘要 · Abstract (English)

Evaluating detailed image captions from Vision-Language Models (VLMs) requires going beyond surface-level semantic similarity. Reference-based metrics (e.g., CIDEr and SPICE) and LLM-as-scorer protocols struggle to verify dense factual claims, while existing QA-based alternatives generally offer lower probe density, narrower domain coverage, or no explicit alignment between individual questions and segmented image regions. We introduce CapProbe, a full-scene dense QA benchmark that turns detailed caption evaluation into region-aligned factual checking. Each image is decomposed into coarse semantic regions covering both foreground and background elements; for every retained region, we generate multiple-choice questions spanning 10 semantic categories, forming a dense checklist of probed visual facts. Guided by a two-tier taxonomy of 37 L1 domains and 219 L2 sub-domains, CapProbe comprises 346 images, 1,868 regions, and 25,650 questions, averaging 74 QA pairs per image. A language judge answers from the caption alone; an Uncertain option and Effective Accuracy provide a judge-dependent proxy for distinguishing unanswered probes from incorrectly resolved ones, while density-based metrics penalize verbose yet uninformative captions. The protocol is cost-effective: by converting unconstrained scalar scoring into structured MCQ reading, it reduces open-ended scoring bias while remaining judge-conditioned and yields relatively stable model rankings under a fixed reader. Experiments on 13 VLMs show large Coverage gaps across models, a clear competency-efficiency trade-off, and failure modes that sparse or overlap-based evaluation often misses. The benchmark data, annotations, and evaluation code will be released soon.

图像描述事实核查多选题评测视觉语言模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。