arXiv:2507.18616cs.CVcs.AI2025-07中稿 · ACM Multimedia 202…被引 7

用一图多文映射重配合成图像描述,提升零样本图文生成效果

SynC: Synthetic Image Caption Dataset Refinement with One-to-many Mapping for Zero-shot Image Captioning

  • 通过候选图像检索与循环一致性评分,重新匹配最契合的图文对
  • 在MS-COCO等数据集上显著提升多个ZIC模型性能,达当前最优
  • 适合需要高质量合成数据的零样本图文生成研究者使用

零样本图像描述(ZIC)越来越多地利用文本到图像(T2I)模型生成的合成数据来减少人工标注成本。然而,这些T2I模型常产生与输入描述语义不符的图像(如缺失物体、属性错误),导致合成图文对存在噪声,影响模型训练。现有数据清洗方法主要针对网络爬取数据中的文本噪声,难以应对合成数据中文字准确但图像失真的问题。为此,我们提出SynC框架,专为优化合成图文数据集而设计。不同于传统过滤或重生成,SynC采用一到多映射策略:先为每条描述检索多个相关候选图像,再通过受循环一致性启发的对齐评分器,选择能通过图像到文本检索重新召回原描述的最佳图像。大量实验表明,SynC在标准基准(MS-COCO、Flickr30k、NoCaps)上持续显著提升多种ZIC模型表现,部分场景达到当前最优。SynC为提升ZIC的合成数据质量提供了有效方案。

原文摘要 · Abstract (English)

Zero-shot Image Captioning (ZIC) increasingly utilizes synthetic datasets generated by text-to-image (T2I) models to mitigate the need for costly manual annotation. However, these T2I models often produce images that exhibit semantic misalignments with their corresponding input captions (e.g., missing objects, incorrect attributes), resulting in noisy synthetic image-caption pairs that can hinder model training. Existing dataset pruning techniques are largely designed for removing noisy text in web-crawled data. However, these methods are ill-suited for the distinct challenges of synthetic data, where captions are typically well-formed, but images may be inaccurate representations. To address this gap, we introduce SynC, a novel framework specifically designed to refine synthetic image-caption datasets for ZIC. Instead of conventional filtering or regeneration, SynC focuses on reassigning captions to the most semantically aligned images already present within the synthetic image pool. Our approach employs a one-to-many mapping strategy by initially retrieving multiple relevant candidate images for each caption. We then apply a cycle-consistency-inspired alignment scorer that selects the best image by verifying its ability to retrieve the original caption via image-to-text retrieval. Extensive evaluations demonstrate that SynC consistently and significantly improves performance across various ZIC models on standard benchmarks (MS-COCO, Flickr30k, NoCaps), achieving state-of-the-art results in several scenarios. SynC offers an effective strategy for curating refined synthetic data to enhance ZIC.

图像描述合成数据零样本

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。