首个百万级图文提示数据集,专为图像生成视频研究设计。
TIP-I2V: A Million-Scale Real Text and Image Prompt Dataset for Image-to-Video Generation
- 构建超170万条用户提供的图文提示,覆盖真实场景需求。
- 对比现有数据集,揭示图文提示在语义与结构上的差异。
- 助力模型优化与安全评估,适合视频生成研究者使用。
视频生成模型正重塑内容创作,其中图像到视频模型因可控性强、视觉一致性高而备受关注。然而,这些模型依赖用户提供的文本与图像提示,目前尚无专门针对此类提示的数据集。本文提出TIP-I2V,首个规模超过170万条独特用户提供的图文提示数据集,专用于图像到视频生成研究。同时提供来自五种前沿图像到视频模型的对应生成视频。我们详细描述了该大规模数据集的耗时与成本较高的构建过程,并将TIP-I2V与两个主流提示数据集(VidProM,文本到视频;DiffusionDB,文本到图像)进行对比,揭示其在基础信息与语义层面的显著差异。该数据集推动图像到视频研究发展:研究人员可利用提示分析用户偏好,评估模型多维度性能;也可聚焦图像到视频模型引发的虚假信息风险,提升模型安全性。TIP-I2V所激发的新研究及与现有数据集的差异,凸显了专用图像到视频提示数据集的重要性。项目主页:https://tip-i2v.github.io。
原文摘要 · Abstract (English)
Video generation models are revolutionizing content creation, with image-to-video models drawing increasing attention due to their enhanced controllability, visual consistency, and practical applications. However, despite their popularity, these models rely on user-provided text and image prompts, and there is currently no dedicated dataset for studying these prompts. In this paper, we introduce TIP-I2V, the first large-scale dataset of over 1.70 million unique user-provided Text and Image Prompts specifically for Image-to-Video generation. Additionally, we provide the corresponding generated videos from five state-of-the-art image-to-video models. We begin by outlining the time-consuming and costly process of curating this large-scale dataset. Next, we compare TIP-I2V to two popular prompt datasets, VidProM (text-to-video) and DiffusionDB (text-to-image), highlighting differences in both basic and semantic information. This dataset enables advancements in image-to-video research. For instance, to develop better models, researchers can use the prompts in TIP-I2V to analyze user preferences and evaluate the multi-dimensional performance of their trained models; and to enhance model safety, they may focus on addressing the misinformation issue caused by image-to-video models. The new research inspired by TIP-I2V and the differences with existing datasets emphasize the importance of a specialized image-to-video prompt dataset. The project is available at https://tip-i2v.github.io.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。