梳理136篇论文，系统总结机器学习监控的实践与短板。
Monitoring Machine Learning Systems: A Multivocal Literature Review
- 通过多源文献综述分析136篇研究，覆盖动机、技术与工具
- 发现生产环境中数据漂移等问题普遍存在且难及时察觉
- 适合研究人员和工程师参考，指导监控方案选型与未来研发
背景：动态生产环境使机器学习系统难以持续可靠运行。运行时问题如数据模式或运行上下文变化导致模型性能下降，在生产中常见。监控可实现问题早期发现与缓解，维护用户信任并避免组织损失。目标：本研究旨在全面综述机器学习监控领域的文献。方法：采用多源文献综述（MLR）方法，依据Garousi的规范，分析136篇论文，涵盖四个关键维度：（1）动机、目标与上下文；（2）监测内容、具体技术、度量指标与工具；（3）贡献与效益；（4）当前局限性。我们还讨论了研究中的若干洞察及其对实践与未来研究的启示。结论：本综述识别并总结了机器学习监控的实践与空白，强调正式文献与灰色文献间的异同。研究成果对学术界与产业界均有价值，有助于选择合适解决方案，揭示现有方法局限，并为后续研究与工具开发指明方向。
原文摘要 · Abstract (English)
Context: Dynamic production environments make it challenging to maintain reliable machine learning (ML) systems. Runtime issues, such as changes in data patterns or operating contexts, that degrade model performance are a common occurrence in production settings. Monitoring enables early detection and mitigation of these runtime issues, helping maintain users' trust and prevent unwanted consequences for organizations. Aim: This study aims to provide a comprehensive overview of the ML monitoring literature. Method: We conducted a multivocal literature review (MLR) following the well established guidelines by Garousi to investigate various aspects of ML monitoring approaches in 136 papers. Results: We analyzed selected studies based on four key areas: (1) the motivations, goals, and context; (2) the monitored aspects, specific techniques, metrics, and tools; (3) the contributions and benefits; and (4) the current limitations. We also discuss several insights found in the studies, their implications, and recommendations for future research and practice. Conclusion: Our MLR identifies and summarizes ML monitoring practices and gaps, emphasizing similarities and disconnects between formal and gray literature. Our study is valuable for both academics and practitioners, as it helps select appropriate solutions, highlights limitations in current approaches, and provides future directions for research and tool development.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。