arXiv:2503.20287cs.CV2025-03ICCV被引 62

构建100万条高质量指令视频编辑数据,提升模型精准理解与生成能力。

InsViE-1M: Effective Instruction-based Video Editing with Elaborate Dataset Construction

  • 通过多阶段筛选生成高保真视频编辑三元组,结合GPT-4o自动过滤。
  • 基于100万条数据训练的InsViE模型,在指令遵循和编辑质量上超越当前最优。
  • 适合视频生成、智能编辑方向的研究者和开发者使用。

基于指令的视频编辑可通过自然语言指令实现高效交互式编辑,无需额外输入如掩码或属性。然而,高质量训练三元组(源视频、编辑后视频、指令)的收集极具挑战性。现有数据集普遍包含低分辨率、短时长且数量有限的源视频,编辑质量不理想,制约了模型性能。本文提出一个包含100万条三元组的高质量指令视频编辑数据集InsViE-1M。首先收集高分辨率、高质量的源视频与图像,再设计高效的编辑-筛选流水线构建优质三元组。对每段源视频,利用不同强度的无分类器引导生成其首帧多个编辑样本,并由GPT-4o根据精心设计指南自动筛选;随后将首帧编辑结果传播至后续帧生成完整编辑视频,并进行帧质量与运动一致性二次筛选。同时从高质量图像中生成并筛选多种视频编辑三元组。基于InsViE-1M,我们提出多阶段学习策略训练InsViE模型,逐步提升其指令理解与编辑能力。大量实验表明,本数据集及训练模型显著优于当前最先进方法。代码已开源。

原文摘要 · Abstract (English)

Instruction-based video editing allows effective and interactive editing of videos using only instructions without extra inputs such as masks or attributes. However, collecting high-quality training triplets (source video, edited video, instruction) is a challenging task. Existing datasets mostly consist of low-resolution, short duration, and limited amount of source videos with unsatisfactory editing quality, limiting the performance of trained editing models. In this work, we present a high-quality Instruction-based Video Editing dataset with 1M triplets, namely InsViE-1M. We first curate high-resolution and high-quality source videos and images, then design an effective editing-filtering pipeline to construct high-quality editing triplets for model training. For a source video, we generate multiple edited samples of its first frame with different intensities of classifier-free guidance, which are automatically filtered by GPT-4o with carefully crafted guidelines. The edited first frame is propagated to subsequent frames to produce the edited video, followed by another round of filtering for frame quality and motion evaluation. We also generate and filter a variety of video editing triplets from high-quality images. With the InsViE-1M dataset, we propose a multi-stage learning strategy to train our InsViE model, progressively enhancing its instruction following and editing ability. Extensive experiments demonstrate the advantages of our InsViE-1M dataset and the trained model over state-of-the-art works. Codes are available at \href{https://github.com/langmanbusi/InsViE}{InsViE}.

视频编辑指令学习数据构建多阶段训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。