检验代码缺陷预测模型的可靠性,发现多数模型预测不准,校准后效果不一。
On the calibration of Just-in-time Defect Prediction
- 测试三种即时缺陷预测模型的预测概率准确性
- 发现模型误差率高达2%-35%,严重偏离真实正确率
- 事后校准方法对提升可信度效果不稳定,需谨慎使用
即时缺陷预测(JIT DP)利用机器学习识别易出错的代码提交,帮助质量保障团队更高效分配资源。尽管预测准确率有所提升,但仍存在误报和漏报问题,尤其在资源有限时可能导致资源浪费或缺陷遗漏。为提升实用性,需评估预测结果的可靠性,即计算置信度分数。简单做法是直接使用模型输出的概率作为置信度,但前提是模型必须充分校准——即概率值真实反映预测正确的可能性。然而现代模型普遍存在校准偏差,导致概率失真。本研究评估了三种JIT DP技术的校准状态,发现所有模型均存在不同程度的校准问题,归一化期望误差(ECE)在2%至35%之间。同时,事后校准方法并未一致改善模型校准性能。
原文摘要 · Abstract (English)
Just in time defect prediction (JIT DP) leverages ML to identify defect-prone code commits, enabling quality assurance (QA) teams to allocate resources more efficiently by focusing on commits that are most likely to contain defects. Although JIT DP techniques have introduced improvements in terms of predictive accuracy, they are still susceptible to misclassification errors such as false positives and negatives. This can lead to wasted resources or undetected defects, a particularly critical concern when QA resources are limited. To mitigate these challenges and preserve the practical utility of JIT DP tools, it becomes essential to estimate the reliability of the predictions, i.e., computing confidence scores. Such scores can help practitioners determine the trustworthiness of predictions and thus prioritize them efficiently. A simple approach to computing confidence scores is to extract, alongside each prediction, the corresponding prediction probabilities and use them as indicators of confidence. However, for these probabilities to reliably serve as confidence scores, the predictive model must be well-calibrated. This means that the prediction probabilities must accurately represent the true likelihood of each prediction being correct. Miscalibration, common in modern ML models, distorts probability scores such that they do not align with the actual correctness probability. In this study, we evaluate the calibration of three JIT DP techniques to determine whether and to what extent they exhibit poor calibration. Furthermore, we assess whether post-calibration methods can improve the calibration of existing JIT defect prediction models. Our results reveal that all evaluated JIT DP models exhibit some level of miscalibration, with ECE ranging from 2-35%. Furthermore, post-calibration methods do not consistently improve the calibration.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。