用细粒度问答评估图像描述真实性,提升模型事实核查能力。
CapProbe: Evaluating Detailed Image Captions via Full-Scene Dense Question Answering

- 将图像分区域生成多选题,实现视觉事实逐块验证。
- 平均每图74个问题,覆盖346张图、25650个问答对。
- 适合评估生成式模型的细节准确性,尤其关注事实一致性。
评估视觉语言模型生成的详细图像描述,需超越表面语义相似性。现有基于参考文本的指标(如CIDEr、SPICE)和大模型评分协议难以验证密集的事实性陈述,而现有问答类方法普遍存在探针密度低、领域覆盖窄或问题与图像区域未对齐的问题。我们提出CapProbe,一个全场景稠密问答基准,将详细描述评估转化为区域对齐的事实核查。每张图像被分解为涵盖前景与背景的粗粒度语义区域;每个保留区域生成横跨10个语义类别的多项选择题,形成密集的事实检查清单。基于37个一级领域和219个二级子领域的双层分类体系,该基准包含346张图像、1868个区域、25650个问题,平均每个图像74个问答对。语言评判者仅凭描述作答;通过设置‘不确定’选项与有效准确率,提供依赖评判者的代理指标,以区分未回答与错误回答问题,同时密度相关指标惩罚冗长但无信息量的描述。该流程成本可控:将开放式评分转为结构化多选阅读,在减少开放评分偏差的同时保持评判者依赖性,并在固定读者下获得相对稳定的模型排名。13个VLM实验显示模型间存在显著覆盖率差距,明确呈现能力-效率权衡,且暴露了稀疏或重叠评估常忽略的失败模式。数据集、标注与评估代码即将发布。
原文摘要 · Abstract (English)
Evaluating detailed image captions from Vision-Language Models (VLMs) requires going beyond surface-level semantic similarity. Reference-based metrics (e.g., CIDEr and SPICE) and LLM-as-scorer protocols struggle to verify dense factual claims, while existing QA-based alternatives generally offer lower probe density, narrower domain coverage, or no explicit alignment between individual questions and segmented image regions. We introduce CapProbe, a full-scene dense QA benchmark that turns detailed caption evaluation into region-aligned factual checking. Each image is decomposed into coarse semantic regions covering both foreground and background elements; for every retained region, we generate multiple-choice questions spanning 10 semantic categories, forming a dense checklist of probed visual facts. Guided by a two-tier taxonomy of 37 L1 domains and 219 L2 sub-domains, CapProbe comprises 346 images, 1,868 regions, and 25,650 questions, averaging 74 QA pairs per image. A language judge answers from the caption alone; an Uncertain option and Effective Accuracy provide a judge-dependent proxy for distinguishing unanswered probes from incorrectly resolved ones, while density-based metrics penalize verbose yet uninformative captions. The protocol is cost-effective: by converting unconstrained scalar scoring into structured MCQ reading, it reduces open-ended scoring bias while remaining judge-conditioned and yields relatively stable model rankings under a fixed reader. Experiments on 13 VLMs show large Coverage gaps across models, a clear competency-efficiency trade-off, and failure modes that sparse or overlap-based evaluation often misses. The benchmark data, annotations, and evaluation code will be released soon.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。