用机器学习替代问卷评估系统变更风险,提升准确性和可审计性。
SENTRY: Deterministic, Intelligent Risk Assessment for IT Change Management

- 基于梯度提升树和混合检索生成模型,融合结构化数据与文本语义分析。
- 在企业级数据上达到0.87的ROC AUC和85%准确率,高风险变更检测率提升3.25倍。
- 模型结果可解释,适合金融等强监管领域的自动化风险评估场景。
大型金融机构的技术变更管理依赖于准确、一致且可审计的风险评估。现实中,许多机构仍依赖主观性强、易被操纵的自填问卷,难以区分日常变更与后续引发重大事故的变更。本文提出SENTRY,一个以确定性机器学习流水线替代问卷评分的风险评估平台,采用梯度提升决策树(XGBoost)与混合检索增强生成(RAG)技术。系统结合结构化运维元数据、应用依赖图谱、历史事故记录,以及对历史变更请求的混合语义与词汇检索。检索步骤捕获非结构化变更请求文本中的风险信号,并将其压缩为单一标量特征后输入模型。该设计确保模型可确定性运行,并通过SHAP值实现每项预测的可解释性。在企业级变更数据上评估,SENTRY取得0.87的ROC AUC与85%整体准确率,高风险变更检测率约为现有流程的3.25倍。最后,论文分析了该架构的设计权衡及其对受监管环境中机器学习应用的启示。
原文摘要 · Abstract (English)
Technology change management in large financial institutions depends on risk assessments that are accurate, consistent, and auditable. In practice, many institutions still rely on self-reported questionnaires. Those questionnaires are subjective, easy to game, and poor at separating routine changes from the ones that later trigger major incidents. This paper presents SENTRY, a risk assessment platform that replaces questionnaire-based scoring with a deterministic machine learning pipeline built from gradient-boosted decision trees (XGBoost) and hybrid retrieval-augmented generation (RAG). The system combines structured operational metadata, application dependency graphs, and historical incident records with a hybrid semantic and lexical search over historical change requests. The retrieval step captures the risk signal in unstructured change request text, then compresses that signal into a single scalar feature before model inference. That design keeps the model deterministic and preserves per-prediction explainability via SHAP values. Evaluated on enterprise-scale change data, SENTRY achieves a ROC AUC of 0.87 and 85% overall accuracy, and it detects high-risk changes at roughly 3.25 times the rate of the existing process. We close by examining the architectural trade-offs behind this design and what they imply for the use of machine learning in regulated change management.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。