将图文描述质量拆解为覆盖度与准确性，更精准评估模型表现。
CAPEval: A Decoupled Caption Evaluation across Understanding and Generation
- 把描述质量分为覆盖程度和事实正确性两个维度进行评估。
- 实验证明：理解任务看覆盖度，生成任务看准确性。
- 适合用于优化不同下游任务的图文生成模型选择。
图文描述是多模态理解与文本到图像生成的核心监督信号。然而，以往评估将描述质量视为单一指标，混淆了两个独立属性：(1) 描述覆盖视觉信息的程度；(2) 图像对描述中声明内容的支持可靠性。为此，我们设计了去耦合的评估基准 CAPEval（Coverage And Precision Evaluation），采用人工撰写的真实描述和人工验证的原子检查项。具体地，CAPEval 将描述质量分解为覆盖率（Coverage）和精确率（Precision）。前者衡量描述对真实事实内容的覆盖程度，后者反映描述中所有陈述的事实正确率。我们选取10位描述生成者，并在四个模型族上开展受控端到端实验，仅以描述来源为变量。实证发现，任务依赖性显著：覆盖率与理解性能强相关，而精确率则是生成性能的主要预测因子。该去耦评估范式不仅实现更细粒度的质量诊断，也为针对不同下游任务选择和优化描述生成者提供可操作指导。
原文摘要 · Abstract (English)
Captions serve as a primary supervision signal for both multimodal understanding and text-to-image generation. However, previous evaluations treat the caption quality as a single scalar objective, which conflates two distinct properties: (1) how much visual information a caption covers and (2) how reliably the image supports its stated claims. To this end, we design a decoupled caption evaluation benchmark, CAPEval (Coverage And Precision Evaluation), with human-written ground-truth captions and human-verified atomic checklist items. Specifically, CAPEval decomposes caption quality into Coverage and Precision. The former quantifies how thoroughly a caption covers ground-truth factual content, while the latter reflects the factual correctness rate of all claims expressed in the caption. We select 10 captioners and further conduct controlled downstream end-to-end experiments with them from four model families, where the caption source is the only variable. Empirically, we find a consistent task-dependent dissociation: Coverage serves as the stronger correlate for understanding performance, whereas Precision acts as the dominant predictor for generation performance. This decoupled evaluation paradigm not only delivers a more fine-grained diagnosis of caption quality, but also offers actionable guidance for selecting and optimizing captioners tailored to different downstream tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。