arXiv:2605.21728cs.CVcs.CL2026-05

提出高效无参考图像描述评估模型,兼顾精度与速度。

BEiTScore: Reference-free Image Captioning Evaluation with an Efficient Cross-Encoder Model

论文配图:BEiTScore: Reference-free Image Captioning Evaluation with an Efficient Cross-Encoder Model
图 1 · 摘自论文原文
  • 基于视觉问答模型初始化的轻量级交叉编码器,提升评估效率。
  • 在多场景基准上表现领先,对细微语义错误更敏感。
  • 适合大规模评估、质量感知解码及奖励引导生成任务。

图像描述评估仍面临挑战,随着视觉语言模型生成长篇、上下文丰富的描述,现有评估指标或依赖计算成本高昂的大型语言模型作为评判者,或受限于标准CLIP编码器,如严格的词元限制、缺乏细粒度敏感性或组合泛化能力(将描述视为“词袋”)。本文提出一种新的可学习评估指标,基于轻量级交叉编码器,其权重初始化源自视觉问答模型,兼顾强初始性能与计算效率。训练采用精心设计的数据混合,包含对抗性大语言模型数据增强,以提升模型对细微视觉-语言错误的敏感度。此外,我们构建了一个新基准,用于评估多样化场景下的详细描述评估。实验表明,该指标在保持高效的同时达到当前最优性能,适用于大规模基准测试、质量感知解码或奖励引导生成。

原文摘要 · Abstract (English)

Image captioning evaluation remains a significant challenge, as vision-language models evolve toward more challenging capabilities such as generating long-form and context-rich descriptions. State-of-the-art evaluation metrics involve extensive computational costs associated with the use of Large Language Models (LLMs) as judges, or instead suffer from the limitations of standard CLIP-based encoders, such as strict token limits, lack of fine-grained sensitivity, or lack of compositional generalization by treating captions as ``bags-of-words.'' We propose a new learned metric that tackles the aforementioned challenges, based on a lightweight cross-encoder that is initialized from a visual question-answering model checkpoint, balancing a strong weight initialization with computational efficiency. Our training scheme uses a carefully assembled data mixture for supervised learning, featuring adversarial LLM-based data augmentations to enhance model sensitivity to fine-grained visual-linguistic errors. We also introduce a new benchmark designed to assess detailed captioning evaluation across diverse scenarios. Experimental results demonstrate that the proposed metric achieves state-of-the-art performance while maintaining the efficiency required for large-scale benchmarking, quality-aware decoding, or reward guidance.

图像描述评估方法交叉编码器无参考

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。