Operon用命名维度实现可增量构建的乱序数据处理,提升机器学习数据流水线效率
Operon: Incremental Construction of Ragged Data via Named Dimensions
- 通过命名维度和显式依赖关系建模乱序数据形状变化
- 实测比基线引擎快14.94倍,扩展时接近线性输出速率
- 适合大规模机器学习数据生成,支持容错与异构任务并行
现代数据处理流程常遇到乱序数据:由自然语言处理、科学测量及自主智能体等场景产生的可变长度元素集合。现有工作流引擎缺乏对乱序数据形状与依赖关系的原生支持,迫使用户手动管理复杂索引与依赖追踪。我们提出Operon,一个基于Rust的工作流引擎,通过命名维度与显式依赖关系的新形式化方法解决上述问题。用户在领域特定语言中声明带维度标注的流水线,系统静态验证正确性,运行时根据执行中逐步发现的数据形状动态调度任务。我们形式化了部分形状推理的数学基础,并证明Operon的增量构建算法在并行环境下保证确定性和收敛性。其对部分已知状态的显式建模支持健壮的持久化与恢复机制,每任务多队列架构实现跨异构任务类型的高效并行。实验表明,Operon相比现有引擎减少14.94倍基线开销,且负载扩展时保持近线性端到端输出速率,特别适用于机器学习应用中的大规模数据生成流水线。
原文摘要 · Abstract (English)
Modern data processing workflows frequently encounter ragged data: collections with variable-length elements that arise naturally in domains like natural language processing, scientific measurements, and autonomous AI agents. Existing workflow engines lack native support for tracking the shapes and dependencies inherent to ragged data, forcing users to manage complex indexing and dependency bookkeeping manually. We present Operon, a Rust-based workflow engine that addresses these challenges through a novel formalism of named dimensions with explicit dependency relations. Operon provides a domain-specific language where users declare pipelines with dimension annotations that are statically verified for correctness, while the runtime system dynamically schedules tasks as data shapes are incrementally discovered during execution. We formalize the mathematical foundation for reasoning about partial shapes and prove that Operon's incremental construction algorithm guarantees deterministic and confluent execution in parallel settings. The system's explicit modeling of partially-known states enables robust persistence and recovery mechanisms, while its per-task multi-queue architecture achieves efficient parallelism across heterogeneous task types. Empirical evaluation demonstrates that Operon outperforms an existing workflow engine with 14.94x baseline overhead reduction while maintaining near-linear end-to-end output rates as workloads scale, making it particularly suitable for large-scale data generation pipelines in machine learning applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。