对比五种漂移检测器在持续监控下的误报率,指导生产环境选型。
When Drift Detectors cry Wolf: False Alarm Rates in continuous ML Monitoring

- 在连续监控下评估五种漂移检测器的误报行为。
- 样本量超200后PSI误报率显著下降,其他检测器波动持续但更稳定。
- 提出校正方法并警示灵敏度与稳定性权衡,适合运维和算法工程师参考。
漂移检测是生产机器学习监控的核心组件,通过比较实时数据与参考分布来触发警报。然而,现有研究多关注合成数据偏移下的检测准确率,忽视了持续监控中的误报问题。在实际环境中,模型需长期、多特征监控,微小误报率会累积成频繁告警,引发告警疲劳。本文系统分析了五种常用漂移检测器(PSI、KS、MMD、LSDD、对抗验证)在连续监控下的误报表现。结果表明:PSI对批次大小敏感,在小样本下误报频繁,但当批次超过约200时行为趋于稳定并显著改善;而KS、MMD、LSDD虽存在批次波动,但在低数据场景下仍比PSI更可靠。采用伯姆费罗尼校正可降低误报,但会牺牲真阳性敏感度,印证了漂移检测中稳定性与灵敏度的固有权衡。本研究为生产系统中检测器选型与校准提供实证依据与实用建议。
原文摘要 · Abstract (English)
Drift detection is a core component of production machine learning monitoring systems, where detectors are used to compare incoming data with a reference distribution and trigger alerts when changes occur. However, these detectors are often evaluated in research settings that emphasize detection accuracy under synthetic shifts, while overlooking false alarms under continuous monitoring. In production environments, models are monitored repeatedly over time and across many features, and even small false positive rates can accumulate into frequent alerts, leading to alarm fatigue. We empirically analyze false positive behavior across five commonly used drift detectors: PSI, KS, MMD, LSDD, and adversarial validation. Consistent with existing literature, PSI exhibits strong sensitivity to batch size, producing frequent false alarms at small sample sizes; however, we further observe that its behavior stabilizes and improves substantially once batch sizes exceed approximately 200 samples. In contrast, KS, MMD, and LSDD display persistent fluctuations across batch sizes, while remaining comparatively more reliable than PSI in low-data regimes. Applying a Bonferroni correction reduces false positive rates, but often at the cost of reduced true positive sensitivity, reinforcing the well-known stability - sensitivity trade-off in drift detection. This work provides a systematic comparison of false positive behavior across multiple drift detectors under continuous monitoring conditions. We identify tradeoffs across detector families and provide practical guidelines for selecting and calibrating drift detectors in production ML systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。