提出视频事件预测新任务与模型,更好理解视频中复杂事件逻辑。
EventFormer: A Node-graph Hierarchical Attention Transformer for Action-centric Video Event Prediction
- 构建节点图结构注意力模型,捕捉事件与语义单元间关系。
- 在35K视频、17.8万片段数据集上超越现有方法。
- 适合研究视频理解、事件推理的学者使用。
脚本事件推断旨在基于上下文预测后续事件,是自然语言处理中的挑战性任务,已在实际应用中取得显著成功。然而,人类事件多以视频形式记录和呈现,当前视觉领域相关研究匮乏。为此,我们提出面向动作中心的视频事件预测(AVEP)任务,该任务通过引入更复杂的逻辑和更丰富的语义信息,区别于现有视频预测任务。我们构建了一个大规模结构化数据集,包含约35,000个标注视频和超过178,000个视频片段,基于已有视频事件数据集,提供细粒度标注,其基本单位为多模态事件参数节点,实现更优的视频事件结构化表示。由于事件结构复杂,传统以图像块或帧为输入的视觉模型难以胜任。为此,我们提出EventFormer——一种基于节点图层次注意力机制的视频事件预测模型,可同时捕捉事件及其参数间的关联以及参数间的指代关系。我们在多个SOTA视频预测模型及大型视觉语言模型(LVLMs)上进行了实验,验证了该任务的复杂性与数据集的价值。我们的方法在所有对比模型中表现最优。数据集与代码将公开,支持实验复现与标注分析。
原文摘要 · Abstract (English)
Script event induction, which aims to predict the subsequent event based on the context, is a challenging task in NLP, achieving remarkable success in practical applications. However, human events are mostly recorded and presented in the form of videos rather than scripts, yet there is a lack of related research in the realm of vision. To address this problem, we introduce AVEP (Action-centric Video Event Prediction), a task that distinguishes itself from existing video prediction tasks through its incorporation of more complex logic and richer semantic information. We present a large structured dataset, which consists of about $35K$ annotated videos and more than $178K$ video clips of event, built upon existing video event datasets to support this task. The dataset offers more fine-grained annotations, where the atomic unit is represented as a multimodal event argument node, providing better structured representations of video events. Due to the complexity of event structures, traditional visual models that take patches or frames as input are not well-suited for AVEP. We propose EventFormer, a node-graph hierarchical attention based video event prediction model, which can capture both the relationships between events and their arguments and the coreferencial relationships between arguments. We conducted experiments using several SOTA video prediction models as well as LVLMs on AVEP, demonstrating both the complexity of the task and the value of the dataset. Our approach outperforms all these video prediction models. We will release the dataset and code for replicating the experiments and annotations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。