arXiv:2512.23977cs.LGcs.AI2025-12

让机器学习模型在流数据中稳定运行,无需改代码。

Causify DataFlow: A Framework For High-performance Machine Learning Stream Computing

  • 用有向无环图+时间窗口保证输出仅依赖历史数据,杜绝未来信息泄露。
  • 批量训练模型可直接用于流式部署,结果一致且无需重写代码。
  • 支持灵活配置频率与内存占用,适合金融、物联网等实时场景。

我们提出DataFlow,一个用于构建、测试和部署高性能机器学习流式计算系统的框架。传统数据科学流程假设数据集有限,从批处理原型迁移到流式生产系统需大量重写代码,导致因果关系破坏、批次边界干扰及实时故障难以复现。DataFlow通过基于有向无环图(DAG)的统一执行模型解决该问题,确保任意时刻t的输出仅依赖于t之前固定长度的历史上下文窗口。该机制保证了批量开发的模型在流式环境中运行时结果一致,无需代码修改。框架通过自动追踪知识时间,严格保障因果性,消除未来信息窥探类错误。支持跨时间与特征维度的灵活分块,仅通过配置即可实现不同频率与内存开销下的模型运行。原生集成Python数据科学生态,提供fit/predict语义,支持在线学习、缓存与增量计算,并通过DAG调度实现自动并行化。我们在金融交易、物联网、欺诈检测与实时分析等场景验证了其有效性。

原文摘要 · Abstract (English)

We present DataFlow, a computational framework for building, testing, and deploying high-performance machine learning systems on unbounded time-series data. Traditional data science workflows assume finite datasets and require substantial reimplementation when moving from batch prototypes to streaming production systems. This gap introduces causality violations, batch boundary artifacts, and poor reproducibility of real-time failures. DataFlow resolves these issues through a unified execution model based on directed acyclic graphs (DAGs) with point-in-time idempotency: outputs at any time t depend only on a fixed-length context window preceding t. This guarantee ensures that models developed in batch mode execute identically in streaming production without code changes. The framework enforces strict causality by automatically tracking knowledge time across all transformations, eliminating future-peeking bugs. DataFlow supports flexible tiling across temporal and feature dimensions, allowing the same model to operate at different frequencies and memory profiles via configuration alone. It integrates natively with the Python data science stack and provides fit/predict semantics for online learning, caching and incremental computation, and automatic parallelization through DAG-based scheduling. We demonstrate its effectiveness across domains including financial trading, IoT, fraud detection, and real-time analytics.

流式计算因果建模DAG调度在线学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。