用合成数据+主动学习,让企业安全团队零门槛用上高精度威胁检测。
Democratizing ML for Enterprise Security: A Self-Sustained Attack Detection Framework
- 先用宽松YARA规则粗筛,再用机器学习去噪,双阶段过滤提升效率。
- 日处理2500亿条日志,最终仅生成少量人工需查工单,准确率随时间持续提升。
- 无需标注数据,安全员可当“老师”指导模型,适合资源有限的团队。
尽管机器学习在安全领域取得进展,但安全运营中心仍普遍依赖规则系统,因ML方案存在资源消耗大、技能门槛高等问题。传统规则方法虽高效,却因僵化导致误报或漏报频发,需持续人工维护。本文提出一种两阶段混合框架,实现机器学习驱动的威胁检测民主化。第一阶段采用有意宽松的YARA规则进行粗粒度筛选,以高召回率为目标;第二阶段使用机器学习分类器剔除第一阶段的误报。为应对数据稀缺,系统引入Simula——一种无需种子的合成数据生成框架,使安全分析师无需数据科学背景即可构建高质量训练集。通过实时调查反馈构建持续优化的闭环,动态调整模型,防止规则退化。该结合主动学习的模型已在生产环境长期测试,覆盖数万系统。系统日均处理原始日志达2500亿条,经两级过滤后仅剩少数工单供人工研判。长期实验表明,得益于主动学习机制,模型精度随时间稳步提升。该方案具备自维持、低开销、低维护特点,使安全人员可作为专家‘教师’引导模型进化。
原文摘要 · Abstract (English)
Despite advancements in machine learning for security, rule-based detection remains prevalent in Security Operations Centers due to the resource intensiveness and skill gap associated with ML solutions. While traditional rule-based methods offer efficiency, their rigidity leads to high false positives or negatives and requires continuous manual maintenance. This paper proposes a novel, two-stage hybrid framework to democratize ML-based threat detection. The first stage employs intentionally loose YARA rules for coarse-grained filtering, optimized for high recall. The second stage utilizes an ML classifier to filter out false positives from the first stage's output. To overcome data scarcity, the system leverages Simula, a seedless synthetic data generation framework, enabling security analysts to create high-quality training datasets without extensive data science expertise or pre-labeled examples. A continuous feedback loop incorporates real-time investigation results to adaptively tune the ML model, preventing rule degradation. This proposed model with active learning has been rigorously tested for a prolonged time in a production environment spanning tens of thousands of systems. The system handles initial raw log volumes often reaching 250 billion events per day, significantly reducing them through filtering and ML inference to a handful of daily tickets for human investigation. Live experiments over an extended timeline demonstrate a general improvement in the model's precision over time due to the active learning feature. This approach offers a self-sustained, low-overhead, and low-maintenance solution, allowing security professionals to guide model learning as expert ``teachers''.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。