无需训练即可自动标注100万条视频步骤，提升长时视频理解能力。
DenseStep2M: A Scalable, Training-Free Pipeline for Dense Instructional Video Annotation
- 用多模态大模型自动分割视频并生成时间对齐的步骤
- 产出100K视频、200万条高质量指令步骤数据集
- 适合做视频理解、步骤定位和跨模态检索的研究者使用
长期视频理解需要解析复杂的时序事件并推理过程性活动。尽管像HowTo100M这样的教学视频语料库为模型训练提供了丰富资源，但存在语音识别噪声大、解说与视觉内容时间对齐不一致等问题。本文提出一种自动化、无需训练的流水线，从真实场景教学视频中提取高质量过程性标注。该方法将视频分段为连贯镜头，过滤对齐差的内容，并利用先进的多模态大模型（Qwen2.5-VL和DeepSeek-R1）生成结构化、时间锚定的步骤。由此构建的DenseStep2M数据集包含约10万视频和200万条详细教学步骤，旨在支持全面的长时视频理解。我们还构建了DenseCaption100基准，包含高质量人工编写字幕。评估显示自动生成步骤与人工标注高度一致。此外，我们在三个核心下游任务上验证了DenseStep2M的有效性：密集视频字幕生成、步骤定位和跨模态检索。在DenseStep2M上微调的模型在字幕质量和时间定位上均有显著提升，并在第一人称、第三人称及混合视角场景中表现出强零样本泛化能力。结果表明DenseStep2M能有效促进多模态对齐与长期活动推理。数据集已公开于https://huggingface.co/datasets/mingjige/DenseStep2M。
原文摘要 · Abstract (English)
Long-term video understanding requires interpreting complex temporal events and reasoning over procedural activities. While instructional video corpora, like HowTo100M, offer rich resources for model training, they present significant challenges, including noisy ASR transcripts and inconsistent temporal alignments between narration and visual content. In this work, we introduce an automated, training-free pipeline to extract high-quality procedural annotations from in-the-wild instructional videos. Our approach segments videos into coherent shots, filters poorly aligned content, and leverages state-of-the-art multimodal and large language models (Qwen2.5-VL and DeepSeek-R1) to generate structured, temporally grounded procedural steps. This pipeline yields DenseStep2M, a large-scale dataset comprising approximately 100K videos and 2M detailed instructional steps, designed to support comprehensive long-form video understanding. To rigorously evaluate our pipeline, we curate DenseCaption100, a benchmark of high-quality, human-written captions. Evaluations demonstrate strong alignment between our auto-generated steps and human annotations. Furthermore, we validate the utility of DenseStep2M across three core downstream tasks: dense video captioning, procedural step grounding, and cross-modal retrieval. Models fine-tuned on DenseStep2M achieve substantial gains in captioning quality and temporal localization, while exhibiting robust zero-shot generalization across egocentric, exocentric, and mixed-perspective domains. These results underscore the effectiveness of DenseStep2M in facilitating advanced multimodal alignment and long-term activity reasoning. Our dataset is available at https://huggingface.co/datasets/mingjige/DenseStep2M.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。