通过重对齐已有描述文本，生成更丰富的图像描述数据。
Altogether: Image Captioning via Re-aligning Alt-text
- 基于现有替代文本，多轮重对齐生成新描述。
- 新数据使图像描述更丰富，提升图文生成与零样本分类性能。
- 适合图像描述、图文生成方向研究者使用。
本文聚焦于利用合成数据提升图像描述质量。现有方法存在两大缺陷：一是从零开始描述图像，忽略已有的替代文本（alt-text）元数据；二是若使用 GPT 等模型作为训练数据源，缺乏透明性。为此，本文提出一种名为 Altogether 的系统性方法，核心思想是编辑并重对齐与图像关联的现有替代文本。我们通过人工标注构建训练数据，标注者以原始替代文本为基础，在多轮迭代中将其与图像内容重新对齐，从而生成包含丰富视觉概念的描述。这区别于以往仅依赖图像和标注者知识的一次性描述任务。我们在此数据上训练一个描述生成器，可规模化模拟替代文本的重对齐过程。实验表明，Altogether 方法生成的描述更丰富，并在文本到图像生成和零样本图像分类任务中均取得提升。
原文摘要 · Abstract (English)
This paper focuses on creating synthetic data to improve the quality of image captions. Existing works typically have two shortcomings. First, they caption images from scratch, ignoring existing alt-text metadata, and second, lack transparency if the captioners' training data (e.g. GPT) is unknown. In this paper, we study a principled approach Altogether based on the key idea to edit and re-align existing alt-texts associated with the images. To generate training data, we perform human annotation where annotators start with the existing alt-text and re-align it to the image content in multiple rounds, consequently constructing captions with rich visual concepts. This differs from prior work that carries out human annotation as a one-time description task solely based on images and annotator knowledge. We train a captioner on this data that generalizes the process of re-aligning alt-texts at scale. Our results show our Altogether approach leads to richer image captions that also improve text-to-image generation and zero-shot image classification tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。