利用动作层级结构实现零样本步骤定位,无需标注即可精准识别操作流程
HiERO-StepG @ Ego4D Step Grounding Challenge: hierarchical activity understanding enables zero-shot step grounding

- 基于弱监督表示学习构建动作功能关联的特征空间,通过聚类识别步骤
- 在Ego4D挑战中达56.27% R@1(IoU=0.3),排名第二且完全零样本
- 适合无标注数据场景下的视频流程理解,尤其适用于动作结构化分析
程序性活动具有明确的层次结构:无论是烹饪食谱还是汽车维修,都天然可分解为步骤与子步骤。传统步骤定位方法依赖大量标注,扩展性差。我们提出,这种层次结构可通过人类活动视频中的重复出现的动作共现模式自然生成。本方法基于HiERO——一种仅需细粒度动作描述的弱监督表征学习方法,将功能相关的动作映射至相近特征空间。在此空间中,仅通过简单聚类即可检测步骤,无需任务特定微调。针对Ego4D步骤定位挑战,我们进一步增强步骤分配在细粒度与粗粒度层面的一致性,强制步骤时间单调性,并后处理以降低噪声预测影响。该方法称为HiERO-StepG,提交时在全局排行榜上以56.27% R@1(IoU=0.3)的成绩位列第二,且完全零样本、无需程序特定标注。
原文摘要 · Abstract (English)
Procedural activities follow well-defined structures: whether we consider a cooking recipe or a mechanic repairing a car, these activities naturally decompose in a hierarchy of steps and sub-steps. Traditional approaches for step grounding require extensive annotations and scale poorly. Instead, we argue that such hierarchical structure can emerge naturally from uncurated videos of human activities through recurring patterns of co-occurring actions and activities. Our approach builds on HiERO, a weakly-supervised representation learning approach that maps close in the feature space actions that are functionally related to each other, leveraging only fine-grained action-level narrations. In this feature space, procedure steps can be detected by a simple clustering, with no additional task-specific fine-tuning. For the Ego4D Step Grounding challenge, we augment this approach by ensuring fine and coarse level agreement in step assignments, enforcing strict temporal monotonicity of the grounded steps and post-processing the detected steps to reduce the impact of noisy predictions. We call this approach HiERO-StepG and it achieves 56.27 % on the R@1 (IoU = 0.3) metric on the global leaderboard at submission time, ranking second while being completely zero-shot and not requiring procedure-specific annotations. Project page: https://github.com/andreazenotto/HiERO-StepG.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。