让科研设施轻松部署和运维机器学习系统
SMOCS: A Streaming Framework for Simplified Deployment, Monitoring, and Optimization of ML Systems in Production
- 基于Kafka的分层抽象，分离基础设施与应用逻辑
- 三线程架构实现数据流、训练与推理解耦，支持持续在线学习
- 配置驱动部署，非工程师也能操作ML流水线
机器学习在科学设施的实时监控、优化与控制中展现出巨大潜力，但其在实际运行环境中的部署与维护仍面临重大工程挑战。各设施的数据协议、格式及基础设施各异，导致每次新应用都需重造集成管道。本文提出SMOCS（Streaming Monitoring Optimization and Control System），一个基于Kafka的容器化框架，包含三项贡献：1）对Apache Kafka的分层抽象，实现基础设施与应用逻辑分离；2）三线程代理架构，时序解耦数据摄入、模型训练与实时推理，支持从实时数据流中持续在线学习；3）配置驱动的部署模式，使领域专家无需软件工程知识即可操作ML流水线。SMOCS具备平台无关性，设计上故障隔离，通过Docker容器化实现水平扩展。该框架已作为开源软件发布于杰斐逊实验室Github。
原文摘要 · Abstract (English)
Machine learning has demonstrated significant potential for real-time monitoring, optimization, and control of scientific facilities. However, deploying and maintaining ML models in operational environments remains a substantial engineering challenge. Each facility presents unique data protocols, non-standard formats, and infrastructure constraints, forcing teams to rebuild integration pipelines for every new application. We present SMOCS (Streaming Monitoring Optimization and Control System), a Kafka-based containerized framework that addresses this challenge through three contributions: 1) a layered abstraction over Apache Kafka that separates infrastructure from application logic, 2) a three-thread agent architecture that temporally decouples data ingestion, model training, and real-time inference enabling continuous online learning from live data streams, and 3) a configuration-driven deployment model that enables domain experts to operate ML pipelines without software engineering expertise. SMOCS is facility platform-agnostic, fault-isolated by design, and horizontally scalable through Docker containerization. The framework is publicly available as open-source software on the Jefferson Lab Github.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。