提出无需大量标注的模型无关概念漂移检测方法,提升工业级ML系统可靠性。
Time to Retrain? Detecting Concept Drifts in Machine Learning Systems
- 提出无监督的CDSeer方法,不依赖特定模型架构且大幅降低标注需求。
- 在8个数据集上表现优于现有最优方法,精度提升57.1%、标签使用减少99%。
- 适合工业场景快速部署,特别适合缺乏标注资源的ML运维团队。
随着机器学习技术的发展,软件工程师广泛使用ML系统处理海量流数据,用于AIOps中的故障预测等任务。然而,基于历史数据训练的模型在生产环境中会因概念漂移(即数据分布及变量间关系的变化)导致性能下降。因此,及时检测概念漂移并重新训练模型至关重要。本文在工业级真实与合成数据集上评估了当前最先进的半监督概念漂移检测方法,发现其不仅需要大量人工标注,且仅适用于特定模型。为克服上述局限,我们提出一种新型模型无关检测方法(CDSeer)。实验表明,CDSeer在8个不同领域数据集上的精度和召回率均优于现有最优方法,且在内部工业数据集上的部署中实现57.1%的精度提升,同时仅需99%更少的标注数据。其性能接近需100%标注的监督方法,显著提升了工业级ML系统的可靠性和可维护性。
原文摘要 · Abstract (English)
With the boom of machine learning (ML) techniques, software practitioners build ML systems to process the massive volume of streaming data for diverse software engineering tasks such as failure prediction in AIOps. Trained using historical data, such ML models encounter performance degradation caused by concept drift, i.e., data and inter-relationship (concept) changes between training and production. It is essential to use concept rift detection to monitor the deployed ML models and re-train the ML models when needed. In this work, we explore applying state-of-the-art (SOTA) concept drift detection techniques on synthetic and real-world datasets in an industrial setting. Such an industrial setting requires minimal manual effort in labeling and maximal generality in ML model architecture. We find that current SOTA semi-supervised methods not only require significant labeling effort but also only work for certain types of ML models. To overcome such limitations, we propose a novel model-agnostic technique (CDSeer) for detecting concept drift. Our evaluation shows that CDSeer has better precision and recall compared to the state-of-the-art while requiring significantly less manual labeling. We demonstrate the effectiveness of CDSeer at concept drift detection by evaluating it on eight datasets from different domains and use cases. Results from internal deployment of CDSeer on an industrial proprietary dataset show a 57.1% improvement in precision while using 99% fewer labels compared to the SOTA concept drift detection method. The performance is also comparable to the supervised concept drift detection method, which requires 100% of the data to be labeled. The improved performance and ease of adoption of CDSeer are valuable in making ML systems more reliable.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。