根据场景图和文字指令生成连贯的步骤图像序列。
ShowHowTo: Generating Scene-Conditioned Step-by-Step Visual Instructions
- 从百万视频自动构建60万组图文数据集，解决训练数据稀缺问题。
- 训练视频扩散模型，生成与输入场景一致的多步图像序列。
- 在步骤、场景和任务准确性上均达当前最佳，适合机器人视觉教学。
本研究旨在根据给定场景图像和一系列文本指令，生成以图像序列为形式的分步视觉指导。该任务极具挑战性，需在特定环境中生成多步骤图像序列以达成复杂目标，而现有大规模训练数据匮乏。为此，本文贡献有三：其一，提出一种自动从教学视频中收集大规模分步视觉指导数据的方法，基于一百万条视频构建了包含60万组图像-文本对的高质量数据集；其二，开发并训练了名为ShowHowTo的视频扩散模型，可生成与输入图像一致的分步视觉指令；其三，在步骤、场景和任务三个维度上评估生成结果，证明模型在各项指标上均达到当前最优表现。代码、数据集及训练模型均已公开。
原文摘要 · Abstract (English)
The goal of this work is to generate step-by-step visual instructions in the form of a sequence of images, given an input image that provides the scene context and the sequence of textual instructions. This is a challenging problem as it requires generating multi-step image sequences to achieve a complex goal while being grounded in a specific environment. Part of the challenge stems from the lack of large-scale training data for this problem. The contribution of this work is thus three-fold. First, we introduce an automatic approach for collecting large step-by-step visual instruction training data from instructional videos. We apply this approach to one million videos and create a large-scale, high-quality dataset of 0.6M sequences of image-text pairs. Second, we develop and train ShowHowTo, a video diffusion model capable of generating step-by-step visual instructions consistent with the provided input image. Third, we evaluate the generated image sequences across three dimensions of accuracy (step, scene, and task) and show our model achieves state-of-the-art results on all of them. Our code, dataset, and trained models are publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。