研究发现ROC AUC比其他指标更稳定,受异常值比例影响小。
Exploring the Impact of Outlier Variability on Anomaly Detection Evaluation Metrics
- 对比三种常用评估指标在不同异常比例下的表现
- F1和AUCPR随异常比例变化而波动,ROC AUC保持稳定
- 固定测试集异常比例时,ROC AUC与AUCPR结果趋同
异常检测领域中,评估指标的选择与解读至关重要,尤其在异常比例变化的场景下。本研究系统分析了三种广泛使用的指标——F1分数、ROC曲线下面积(ROC AUC)和精确率-召回率曲线下面积(AUCPR)——在不同异常比例和污染阈值条件下的行为表现。通过涵盖多个主流异常检测算法的全面实验,研究揭示:尽管F1分数和AUCPR对异常比例敏感,但ROC AUC在各类条件下均保持一致性,不受异常比例波动影响。此外,在测试集中异常比例固定的情况下,ROC AUC与AUCPR呈现高度一致,表明二者选择差异在此类场景下影响较小。研究为异常检测中的指标选取与解读提供了更精细的理解,对研究人员与实践者具有重要参考价值。
原文摘要 · Abstract (English)
Anomaly detection is a dynamic field, in which the evaluation of models plays a critical role in understanding their effectiveness. The selection and interpretation of the evaluation metrics are pivotal, particularly in scenarios with varying amounts of anomalies. This study focuses on examining the behaviors of three widely used anomaly detection metrics under different conditions: F1 score, Receiver Operating Characteristic Area Under Curve (ROC AUC), and Precision-Recall Curve Area Under Curve (AUCPR). Our study critically analyzes the extent to which these metrics provide reliable and distinct insights into model performance, especially considering varying levels of outlier fractions and contamination thresholds in datasets. Through a comprehensive experimental setup involving widely recognized algorithms for anomaly detection, we present findings that challenge the conventional understanding of these metrics and reveal nuanced behaviors under varying conditions. We demonstrated that while the F1 score and AUCPR are sensitive to outlier fractions, the ROC AUC maintains consistency and is unaffected by such variability. Additionally, under conditions of a fixed outlier fraction in the test set, we observe an alignment between ROC AUC and AUCPR, indicating that the choice between these two metrics may be less critical in such scenarios. The results of our study contribute to a more refined understanding of metric selection and interpretation in anomaly detection, offering valuable insights for both researchers and practitioners in the field.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。