用图神经网络识别第一人称视频中的关键步骤，效果显著优于现有方法。
Keystep Recognition using Graph Neural Networks
- 将视频片段建模为节点，构建稀疏图来捕捉长期依赖关系。
- 在Ego-Exo4D数据集上准确率明显领先现有方法，且计算效率更高。
- 融合第三人称视频或自动字幕作为额外信息，提升模型泛化能力。
我们将关键步骤识别问题视为节点分类任务，提出一种灵活的图学习框架GLEVR，用于细粒度的关键步骤识别，能有效利用第一人称视频中的长期依赖。该方法将每个第一人称视频片段作为图中的一个节点，构建稀疏图结构，具有计算高效性，性能远超现有更大规模模型。训练时，我们引入第一人称与第三人称视频间的对齐信息，并加入自动字幕作为额外模态，将每个第三人称视频片段（如可用）或视频字幕作为训练时的附加节点。我们探索了多种节点间连接策略。在Ego-Exo4D数据集上进行了大量实验，结果表明，所提出的图基框架显著优于现有方法。
原文摘要 · Abstract (English)
We pose keystep recognition as a node classification task, and propose a flexible graph-learning framework for fine-grained keystep recognition that is able to effectively leverage long-term dependencies in egocentric videos. Our approach, termed GLEVR, consists of constructing a graph where each video clip of the egocentric video corresponds to a node. The constructed graphs are sparse and computationally efficient, outperforming existing larger models substantially. We further leverage alignment between egocentric and exocentric videos during training for improved inference on egocentric videos, as well as adding automatic captioning as an additional modality. We consider each clip of each exocentric video (if available) or video captions as additional nodes during training. We examine several strategies to define connections across these nodes. We perform extensive experiments on the Ego-Exo4D dataset and show that our proposed flexible graph-based framework notably outperforms existing methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。