arXiv:2501.08591cs.DBcs.AI2025-01被引 2

OpenMLDB统一在线离线特征计算,实现毫秒级实时更新。

OpenMLDB: A Real-Time Relational Data Feature Computation System for Online ML

  • 用统一查询计划保证离线与在线特征计算一致
  • 通过预聚合和自适应调整提升长窗口计算速度
  • 适合需要实时特征的推荐、风控等系统使用

高效的特征计算对众多在线机器学习应用至关重要。传统上,特征计算分为离线训练和在线服务两个阶段,常依赖不同接口语言和实现方式,导致结果不一致。许多在线特征涉及复杂的时间序列计算(如变长表窗函数),不同于标准流式或分析查询,现有系统(如Spark、Flink、DuckDB)在这些场景下通常产生数秒延迟,难以满足实时在线机器学习对及时特征更新的需求。本文提出OpenMLDB,已在4Paradigm的SageOne平台及100多个真实场景中部署。技术上,OpenMLDB首先采用统一查询计划生成器,确保离线与在线阶段结果一致,显著降低特征部署开销;其次提供在线执行引擎,通过预聚合解决长窗口计算瓶颈,通过数据自适应机制优化多表窗口合并性能;同时具备支持窗口并行优化和时间感知数据倾斜处理的高性能离线引擎;第三,采用紧凑数据格式和面向流的索引策略,最大化内存利用率并加速数据访问。测试与真实负载评估显示,相比基线系统有显著性能提升和资源节省。目前OpenMLDB开源社区已有超过150名贡献者,GitHub获1.6k星标。

原文摘要 · Abstract (English)

Efficient and consistent feature computation is crucial for a wide range of online ML applications. Typically, feature computation is divided into two distinct phases, i.e., offline stage for model training and online stage for model serving. These phases often rely on execution engines with different interface languages and function implementations, causing significant inconsistencies. Moreover, many online ML features involve complex time-series computations (e.g., functions over varied-length table windows) that differ from standard streaming and analytical queries. Existing data processing systems (e.g., Spark, Flink, DuckDB) often incur multi-second latencies for these computations, making them unsuitable for real-time online ML applications that demand timely feature updates. This paper presents OpenMLDB, a feature computation system deployed in 4Paradigm's SageOne platform and over 100 real scenarios. Technically, OpenMLDB first employs a unified query plan generator for consistent computation results across the offline and online stages, significantly reducing feature deployment overhead. Second, OpenMLDB provides an online execution engine that resolves performance bottlenecks caused by long window computations (via pre-aggregation) and multi-table window unions (via data self-adjusting). It also provides a high-performance offline execution engine with window parallel optimization and time-aware data skew resolving. Third, OpenMLDB features a compact data format and stream-focused indexing to maximize memory usage and accelerate data access. Evaluations in testing and real workloads reveal significant performance improvements and resource savings compared to the baseline systems. The open community of OpenMLDB now has over 150 contributors and gained 1.6k stars on GitHub.

特征工程实时计算数据系统机器学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。