Meta用健康检查系统保障大规模持续部署的安全性
Making Deployments Safe at Meta: Health Checks for Continuous Change-Safety
- 构建分布式健康检查系统，通过模板化指标与阈值预防故障
- 支持分层渐进发布，发现问题自动触发回滚
- 解决告警噪声等问题，探索AI辅助调优方向
向大规模生产系统持续部署带来发布速度与可靠性的矛盾。每次变更都可能引发可靠性问题，但延迟又会错失机会。本文介绍Meta用于调节这一矛盾的部署健康检查基础设施，涵盖名为Service Health Checker的预防型分布式系统架构。该系统允许检查作者组合模板化指标查询、阈值和工作流谓词；并集成到分层与分阶段发布流程中，使回归问题能自动触发回滚。随后讨论了规模化运行中出现的操作问题，如噪声、告警疲劳、指标漂移和未覆盖的回归，并介绍了通过度量、工具改进和默认策略优化来应对的方案。最后总结了多年运维健康检查的经验教训，以及下一步探索方向，包括AI辅助健康检查调优。
原文摘要 · Abstract (English)
Continuous deployment to large scale production systems creates a tension between release velocity and reliability. Every change is a potential reliability incident, yet every delay is a missed opportunity. This paper describes the deployment time health check infrastructure that Meta uses to mediate this tension across thousands of heterogeneous services. We summarize the architecture of this prevention based distributed system's service called Service Health Checker, explain how check authors compose templated metric queries, thresholds, and workflow predicates; and discuss how the system is integrated with tiered and phased rollouts so that regressions trigger automatic rollback. We then describe the operational problems that emerged at scale, such as noise, alert fatigue, drift, and uncovered regressions, and the program of measurement, tooling, and improved defaults we deployed to address them. We close with lessons learned from years of operating deployment health checks at Meta, and the directions we are exploring next, including AI assisted health check tuning. Index Terms: deployment safety, continuous deployment, monitoring, software reliability, release engineering, software reliability engineering, AIOps, anomaly detection
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。