arXiv:2510.23907cs.CVcs.AI2025-10中稿 · NeurIPS

动态调整采样窗口,让视频指令生成更连贯

DynaStride: Dynamic Stride Windowing with MMCoT for Instructional Multi-Scene Captioning

  • 自适应帧采样+多模态思维链,捕捉场景关键变化
  • 在YouCookII上优于VLLaMA3和GPT-4o,BLEU/METEOR提升1.5~3.2点
  • 适合需要精准动作描述的教育类视频生成任务

instructional视频的场景级字幕生成能通过理解视觉线索与时间结构来增强学习效果。通过将视觉信息与文本引导对齐,这种理解支持程序性学习和多模态推理,为技能习得提供更丰富的上下文。然而,未能捕捉时间结构的字幕可能缺乏连贯性和质量,造成困惑并削弱视频的教育意图。为此,我们提出DynaStride,一个无需人工场景分割即可生成连贯场景级字幕的流水线。基于YouCookII数据集的场景标注,DynaStride采用自适应帧采样和多模态窗口化技术,捕获每个场景内的关键过渡。随后,利用多模态链式思考(MMCoT)生成多个动作-对象对,并通过动态步长窗口选择算法融合,平衡时间上下文与冗余。最终的场景级字幕整合了视觉语义与时间推理,形成单一指令性描述。实证评估显示,相比VLLaMA3和GPT-4o等强基线,在N-gram指标(BLEU、METEOR)和语义相似度(BERTScore、CLIPScore)上均有稳定提升。定性分析表明,DynaStride生成的字幕更具时间连贯性和信息量,为提升AI驱动的教育内容生成提供了新方向。

原文摘要 · Abstract (English)

Scene-level captioning in instructional videos can enhance learning by requiring an understanding of both visual cues and temporal structure. By aligning visual cues with textual guidance, this understanding supports procedural learning and multimodal reasoning, providing a richer context for skill acquisition. However, captions that fail to capture this structure may lack coherence and quality, which can create confusion and undermine the video's educational intent. To address this gap, we introduce DynaStride, a pipeline to generate coherent, scene-level captions without requiring manual scene segmentation. Using the YouCookII dataset's scene annotations, DynaStride performs adaptive frame sampling and multimodal windowing to capture key transitions within each scene. It then employs a multimodal chain-of-thought process to produce multiple action-object pairs, which are refined and fused using a dynamic stride window selection algorithm that adaptively balances temporal context and redundancy. The final scene-level caption integrates visual semantics and temporal reasoning in a single instructional caption. Empirical evaluations against strong baselines, including VLLaMA3 and GPT-4o, demonstrate consistent gains on both N-gram-based metrics (BLEU, METEOR) and semantic similarity measures (BERTScore, CLIPScore). Qualitative analyses further show that DynaStride produces captions that are more temporally coherent and informative, suggesting a promising direction for improving AI-powered instructional content generation.

视频生成多模态指令生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。