从单个视频中提取详细动作流程图，支持并行与顺序步骤识别。
Box2Flow: Instance-based Action Flow Graphs from Videos
- 基于视频边界框和成对边概率，用生成树算法构建流程图。
- 在MM-ReS和YouCookII数据集上准确还原任务步骤关系。
- 适合需要细粒度动作解析的视频理解任务，如教学视频分析。
网络上有大量程序性视频展示如何完成各类任务。这些任务通常有多种实现方式和步骤顺序，部分步骤可并行执行，而另一些则需按特定顺序进行。流程图可用于描述任务步骤间的关系。现有基于任务的方法尝试为某一任务的所有视频学习单一流程图，但提取出的图往往过于抽象，难以捕捉详细步骤信息。本文旨在通过从单个视频中提取流程图，学习准确且丰富的流程图。我们提出Box2Flow，一种基于实例的方法，从给定的程序性视频中预测步骤流程图。具体而言，我们从视频中提取边界框，预测步骤对之间的成对边概率，并使用生成树算法构建流程图。在MM-ReS和YouCookII数据集上的实验表明，该方法能有效提取流程图。
原文摘要 · Abstract (English)
A large amount of procedural videos on the web show how to complete various tasks. These tasks can often be accomplished in different ways and step orderings, with some steps able to be performed simultaneously, while others are constrained to be completed in a specific order. Flow graphs can be used to illustrate the step relationships of a task. Current task-based methods try to learn a single flow graph for all available videos of a specific task. The extracted flow graphs tend to be too abstract, failing to capture detailed step descriptions. In this work, our aim is to learn accurate and rich flow graphs by extracting them from a single video. We propose Box2Flow, an instance-based method to predict a step flow graph from a given procedural video. In detail, we extract bounding boxes from videos, predict pairwise edge probabilities between step pairs, and build the flow graph with a spanning tree algorithm. Experiments on MM-ReS and YouCookII show our method can extract flow graphs effectively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。