用大模型提炼视频步骤,提升动作定位准确率。
Learning to Localize Actions in Instructional Videos with LLM-Based Multi-Pathway Text-Video Alignment
- 用大模型过滤无关解说,提取关键步骤摘要
- 多路径对齐策略融合语义相似性生成可靠匹配
- 在三个任务上超越现有方法,效果提升超2.8%
在缺乏大规模标注视频数据的情况下,定位教学视频中操作步骤的时间边界极具挑战。现有方法通过对比学习对齐视频片段与语音转录文本,但未能处理解说噪声(如无关内容)和时间戳不可靠问题。本文提出一种新训练框架:利用大语言模型(LLM)理解流程并过滤无关信息,将相关步骤摘要为LLM-steps;进一步设计多路径文本-视频对齐(MPTVA)策略,通过三种路径衡量对齐关系:(1) 基于解说时间戳的步骤-解说-视频对齐,(2) 基于长期语义相似性的直接步骤-视频对齐,(3) 基于通用视频领域学习的短时细粒度语义对齐。各路径结果融合生成可靠的伪匹配用于训练。在多个任务和设置下实验表明,该方法在步骤定位、过程步骤定位和解说定位三项任务上分别优于当前最佳方法5.9%、3.1%和2.8%。
原文摘要 · Abstract (English)
Learning to localize temporal boundaries of procedure steps in instructional videos is challenging due to the limited availability of annotated large-scale training videos. Recent works focus on learning the cross-modal alignment between video segments and ASR-transcripted narration texts through contrastive learning. However, these methods fail to account for the alignment noise, i.e., irrelevant narrations to the instructional task in videos and unreliable timestamps in narrations. To address these challenges, this work proposes a novel training framework. Motivated by the strong capabilities of Large Language Models (LLMs) in procedure understanding and text summarization, we first apply an LLM to filter out task-irrelevant information and summarize task-related procedure steps (LLM-steps) from narrations. To further generate reliable pseudo-matching between the LLM-steps and the video for training, we propose the Multi-Pathway Text-Video Alignment (MPTVA) strategy. The key idea is to measure alignment between LLM-steps and videos via multiple pathways, including: (1) step-narration-video alignment using narration timestamps, (2) direct step-to-video alignment based on their long-term semantic similarity, and (3) direct step-to-video alignment focusing on short-term fine-grained semantic similarity learned from general video domains. The results from different pathways are fused to generate reliable pseudo step-video matching. We conducted extensive experiments across various tasks and problem settings to evaluate our proposed method. Our approach surpasses state-of-the-art methods in three downstream tasks: procedure step grounding, step localization, and narration grounding by 5.9\%, 3.1\%, and 2.8\%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。