从纯文本生成图文匹配的封面图，自建数据集提升生成质量
MMCIG: Multimodal Cover Image Generation for Text-only Documents and Its Dataset Construction via Pseudo-labeling
- 用多模态伪标签法，结合图文排名构建高质量数据集
- 相比单模态伪标签，生成图像更准确、与文本匹配度更高
- 适合想做图文生成、数据构建或跨模态理解的研究者
本研究提出一项新的封面图像生成任务：从纯文本文档中生成简洁摘要和视觉对应的图像。由于该任务缺乏现有数据集，我们提出一种多模态伪标签方法，低成本构建高质量数据集。首先收集包含多个图像及对应标题的文档，并剔除事实不一致样本；随后从每篇文档的多张图片中选一张，分别以真实摘要为基准独立对图像和标题进行排序；当某张图像及其标题在各自排名中均位列第一时，标注为伪标签；最后去除文本中直接提及图像的文档。实验表明，所提方法构建的数据集更精确，生成的图像质量优于仅使用文本或图像的伪标签方法。代码已公开于：https://github.com/HyeyeeonKim/MMCIG
原文摘要 · Abstract (English)
In this study, we introduce a novel cover image generation task that produces both a concise summary and a visually corresponding image from a given text-only document. Because no existing datasets are available for this task, we propose a multimodal pseudo-labeling method to construct high-quality datasets at low cost. We first collect documents that contain multiple images with their captions, and their summaries by excluding factually inconsistent instances. Our approach selects one image from the multiple images accompanying the documents. Using the gold summary, we independently rank both the images and their captions. Then, we annotate a pseudo-label for an image when both the image and its corresponding caption are ranked first in their respective rankings. Finally, we remove documents that contain direct image references within texts. Experimental results demonstrate that the proposed multimodal pseudo-labeling method constructs more precise datasets and generates higher quality images than text- and image-only pseudo-labeling methods, which consider captions and images separately. We release our code at: https://github.com/HyeyeeonKim/MMCIG
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。