用少量标注日志实现高效间歇性失败检测,准确率超现有方法。
Efficient Detection of Intermittent Job Failures Using Few-Shot Learning
- 基于小样本学习,仅需少量人工标注日志微调语言模型生成特征嵌入。
- 在所有项目中仅用12个样本即达70%-88%的F1分数,显著优于旧方法。
- 适合缺乏大量标注数据的工业场景,提升实际部署可行性。
持续集成与部署流水线中的间歇性作业失败是开发者的重大挑战,其源于非确定性问题(如脆弱测试或基础设施故障),而非常规代码错误。以往研究依赖大规模日志数据训练机器学习模型进行分类。现有最优方法采用基于重试的启发式规则,但在未明确重试策略的场景下,会将32%的间歇性失败误判为常规失败。本研究通过分析5个工业项目和1个开源项目的2,125个失败案例发现,该问题普遍存在。为此,提出一种基于小样本学习(FSL)的新方法:使用少量人工标注日志微调小型语言模型,生成丰富嵌入特征,进而训练分类器。实验表明,该方法在全部项目中仅需12个样本即可达到70%-88%的F1分数,显著优于现有方法(4个项目中仅为34%-52%)。研究强调数据质量的重要性,并提供更高效、实用的间歇性失败检测框架。
原文摘要 · Abstract (English)
One of the main challenges developers face in the use of continuous integration (CI) and deployment pipelines is the occurrence of intermittent job failures, which result from unexpected non-deterministic issues (e.g., flaky tests or infrastructure problems) rather than regular code-related errors such as bugs. Prior studies developed machine learning (ML) models trained on large datasets of job logs to classify job failures as either intermittent or regular. As an alternative to costly manual labeling of large datasets, the state-of-the-art (SOTA) approach leveraged a heuristic based on non-deterministic job reruns. However, this method mislabels intermittent job failures as regular in contexts where rerunning suspicious job failures is not an explicit policy, and therefore limits the SOTA's performance in practice. In fact, our manual analysis of 2,125 job failures from 5 industrial and 1 open-source projects reveals that, on average, 32% of intermittent job failures are mislabeled as regular. To address these limitations, this paper introduces a novel approach to intermittent job failure detection using few-shot learning (FSL). Specifically, we fine-tune a small language model using a few number of manually labeled log examples to generate rich embeddings, which are then used to train an ML classifier. Our FSL-based approach achieves 70-88% F1-score with only 12 shots in all projects, outperforming the SOTA, which proved ineffective (34-52% F1-score) in 4 projects. Overall, this study underlines the importance of data quality over quantity and provides a more efficient and practical framework for the detection of intermittent job failures in organizations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。