在部分标注数据中同时识别已知与未知动作,提升动作分割准确性。
Looking into the Unknown: Exploring Action Discovery for Segmentation of Known and Unknown Actions
- 用已知动作的时序特征引导未知动作的分割区间。
- 通过嵌入相似性识别未知动作中的语义类别,提升分割精度。
- 适用于神经科学、医疗等标注不全的真实场景。
我们提出动作发现(Action Discovery),一种新的时间动作分割设定,旨在应对模糊动作定义与部分标注数据中的不完整标签问题。训练数据中仅对部分动作(称为已知动作)进行标注,其余未知动作保持未标注状态。该设定在神经科学领域尤为相关,例如行走、进食等明确行为常与细微或罕见行为共存,且因标签模糊或缺失导致数据天然部分标注。为此,我们提出两步法:首先引入粒度引导分割模块(GGSM),通过模仿已知动作的粒度来识别已知与未知动作的时序区间;其次提出未知动作片段分配(UASA),基于学习到的嵌入相似性识别未知动作中的语义有意义类别。我们在三组挑战性数据集(Breakfast、50Salads、Desktop Assembly)上系统验证了该设定,结果表明所提方法显著优于现有基线。
原文摘要 · Abstract (English)
We introduce Action Discovery, a novel setup within Temporal Action Segmentation that addresses the challenge of defining and annotating ambiguous actions and incomplete annotations in partially labeled datasets. In this setup, only a subset of actions - referred to as known actions - is annotated in the training data, while other unknown actions remain unlabeled. This scenario is particularly relevant in domains like neuroscience, where well-defined behaviors (e.g., walking, eating) coexist with subtle or infrequent actions that are often overlooked, as well as in applications where datasets are inherently partially annotated due to ambiguous or missing labels. To address this problem, we propose a two-step approach that leverages the known annotations to guide both the temporal and semantic granularity of unknown action segments. First, we introduce the Granularity-Guided Segmentation Module (GGSM), which identifies temporal intervals for both known and unknown actions by mimicking the granularity of annotated actions. Second, we propose the Unknown Action Segment Assignment (UASA), which identifies semantically meaningful classes within the unknown actions, based on learned embedding similarities. We systematically explore the proposed setting of Action Discovery on three challenging datasets - Breakfast, 50Salads, and Desktop Assembly - demonstrating that our method considerably improves upon existing baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。