无需人工标注,用变形测试发现姿态估计系统漏洞。
Metamorphic Testing for Pose Estimation Systems
- 设计变形测试框架,自动生成测试用例
- 在FLIC和PHOENIX数据集上检出故障率超人工标注
- 可按应用需求定制规则,适配人/动物等不同场景
姿态估计系统广泛应用于体育分析、牲畜护理等领域。由于其潜在影响重大,需系统性测试其行为与失效可能。但传统测试面临“真值难题”及人工标注成本高昂问题,且不同应用场景关注主体(如人或动物)和关键点(如四肢或全身+面部)各异,导致标注数据难复用。为此,本文提出MET-POSE——一种无需人工标注的变形测试框架,可评估系统在多样化条件下的表现。该框架不限于特定应用,提供一组常见计算机视觉挑战的变形规则及其评估方法。实验表明,在Mediapipe Holistic系统上使用FLIC和PHOENIX数据集时,MET-POSE检出故障的能力与人工标注相当甚至更优,且用户可根据自身应用需求灵活调整规则集。
原文摘要 · Abstract (English)
Pose estimation systems are used in a variety of fields, from sports analytics to livestock care. Given their potential impact, it is paramount to systematically test their behaviour and potential for failure. This is a complex task due to the oracle problem and the high cost of manual labelling necessary to build ground truth keypoints. This problem is exacerbated by the fact that different applications require systems to focus on different subjects (e.g., human versus animal) or landmarks (e.g., only extremities versus whole body and face), which makes labelled test data rarely reusable. To combat these problems we propose MET-POSE, a metamorphic testing framework for pose estimation systems that bypasses the need for manual annotation while assessing the performance of these systems under different circumstances. MET-POSE thus allows users of pose estimation systems to assess the systems in conditions that more closely relate to their application without having to label an ad-hoc test dataset or rely only on available datasets, which may not be adapted to their application domain. While we define MET-POSE in general terms, we also present a non-exhaustive list of metamorphic rules that represent common challenges in computer vision applications, as well as a specific way to evaluate these rules. We then experimentally show the effectiveness of MET-POSE by applying it to Mediapipe Holistic, a state of the art human pose estimation system, with the FLIC and PHOENIX datasets. With these experiments, we outline numerous ways in which the outputs of MET-POSE can uncover faults in pose estimation systems at a similar or higher rate than classic testing using hand labelled data, and show that users can tailor the rule set they use to the faults and level of accuracy relevant to their application.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。