arXiv:2411.05706cs.CVcs.CL2024-11被引 16

用文生图模型评估图像描述生成质量,无需人工参考文本。

Image2Text2Image: A Novel Framework for Label-Free Evaluation of Image-to-Text Generation with Text-to-Image Diffusion Models

  • 用文生图模型将自动生成的描述还原成图像
  • 通过原图与重建图的特征相似度判断描述质量
  • 适合无参考文本时的自动评估,尤其适合模型迭代测试

评估自动生成图像描述的质量是一项复杂任务,需涵盖语法、覆盖度、准确性和真实性等多个维度。尽管人工评估提供有价值见解,但成本高且耗时。现有自动指标如BLEU、ROUGE、METEOR和CIDEr与人类判断相关性弱。为此,我们提出Image2Text2Image框架,利用Stable Diffusion或DALL-E等扩散模型进行文生图生成。该框架首先由待评估的图像描述模型生成文本,再用此文本驱动扩散模型生成新图像。通过对比原始图像与生成图像的特征相似度,衡量描述忠实度:相似度高说明描述准确,低则揭示模型缺陷。本方法无需人工标注参考描述,可有效评估图像描述模型。大量实验与人工评估验证了其有效性。代码与数据集将公开以支持社区研究。

原文摘要 · Abstract (English)

Evaluating the quality of automatically generated image descriptions is a complex task that requires metrics capturing various dimensions, such as grammaticality, coverage, accuracy, and truthfulness. Although human evaluation provides valuable insights, its cost and time-consuming nature pose limitations. Existing automated metrics like BLEU, ROUGE, METEOR, and CIDEr attempt to fill this gap, but they often exhibit weak correlations with human judgment. To address this challenge, we propose a novel evaluation framework called Image2Text2Image, which leverages diffusion models, such as Stable Diffusion or DALL-E, for text-to-image generation. In the Image2Text2Image framework, an input image is first processed by a selected image captioning model, chosen for evaluation, to generate a textual description. Using this generated description, a diffusion model then creates a new image. By comparing features extracted from the original and generated images, we measure their similarity using a designated similarity metric. A high similarity score suggests that the model has produced a faithful textual description, while a low score highlights discrepancies, revealing potential weaknesses in the model's performance. Notably, our framework does not rely on human-annotated reference captions, making it a valuable tool for assessing image captioning models. Extensive experiments and human evaluations validate the efficacy of our proposed Image2Text2Image evaluation framework. The code and dataset will be published to support further research in the community.

图像描述自动评估扩散模型无参考

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。