用视觉语言模型重写描述，生成百万级开放词汇图像标注数据集
ROVI: A VLM-LLM Re-Captioned Dataset for Open-Vocabulary Instance-Grounded Text-to-Image Generation
- 先用VLM生成详细视觉描述，再用LLM提取候选类别用于检测
- 数据集含100万张高分辨率图像，类别数量比现有数据多两个数量级
- 适合做实例定位精准的文本生成图像研究，尤其关注细节捕捉
我们提出ROVI，一个用于实例定位文本到图像生成的高质量合成数据集，基于100万张精选网络图像构建。核心创新在于一种称为重标注（re-captioning）的策略，聚焦预检测阶段：使用视觉语言模型（VLM）生成全面的视觉描述，再由大语言模型（LLM）提取潜在类别列表，供开放词汇检测器（OVD）使用。该方法在实例标注基础上生成全局提示，同时捕捉人类常忽略的次要视觉元素。评估显示，ROVI在图像质量与分辨率上优于现有检测数据集，且类别数量多两个数量级，具备开放词汇特性。为验证效果，基于ROVI训练的GLIGEN模型在实例定位精度、提示忠实度和美学质量上显著超越现有先进方法。数据集及可复现流程已公开于https://github.com/CihangPeng/ROVI。
原文摘要 · Abstract (English)
We present ROVI, a high-quality synthetic dataset for instance-grounded text-to-image generation, created by labeling 1M curated web images. Our key innovation is a strategy called re-captioning, focusing on the pre-detection stage, where a VLM (Vision-Language Model) generates comprehensive visual descriptions that are then processed by an LLM (Large Language Model) to extract a flat list of potential categories for OVDs (Open-Vocabulary Detectors) to detect. This approach yields a global prompt inherently linked to instance annotations while capturing secondary visual elements humans typically overlook. Evaluations show that ROVI exceeds existing detection datasets in image quality and resolution while containing two orders of magnitude more categories with an open-vocabulary nature. For demonstrative purposes, a text-to-image model GLIGEN trained on ROVI significantly outperforms state-of-the-art alternatives in instance grounding accuracy, prompt fidelity, and aesthetic quality. Our dataset and reproducible pipeline are available at https://github.com/CihangPeng/ROVI.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。