用大模型自动设计实时数据流水线，效率提升6倍以上。
AutoStreamPipe: LLM Assisted Automatic Generation of Data Stream Processing Pipelines
- 引入思维超图机制，将用户意图转化为可执行流水线
- 在多场景测试中开发耗时降低6.3倍，错误率下降5.19倍
- 适合需要快速构建高可靠流处理系统的开发者
数据流水线在流处理中至关重要，支持实时数据的高效采集、处理与交付。本文提出AutoStreamPipe框架，利用大语言模型（LLM）自动化生成和部署流处理流水线。该框架通过集成扩展版思维超图（HGoT），弥合用户高层次意图与分布式流系统平台间实现的语义鸿沟，支持结构化多智能体推理。结合容错执行策略与高级查询分析，AutoStreamPipe实现了高精度流水线生成。在多样化流水线上的实验表明，相比传统LLM代码生成方法，其开发时间缩短6.3倍，错误率下降5.19倍，基于新提出的无错误评分（EFS）进行评估。
原文摘要 · Abstract (English)
Data pipelines are essential in stream processing as they enable the efficient collection, processing, and delivery of real-time data, supporting rapid data analysis. In this paper, we present AutoStreamPipe, a novel framework that employs Large Language Models (LLMs) to automate the design, generation, and deployment of stream processing pipelines. AutoStreamPipe bridges the semantic gap between high-level user intent and platform-specific implementations across distributed stream processing systems for structured multi-agent reasoning by integrating a Hypergraph of Thoughts (HGoT) as an extended version of GoT. AutoStreamPipe combines resilient execution strategies, advanced query analysis, and HGoT to deliver pipelines with good accuracy. Experimental evaluations on diverse pipelines demonstrate that AutoStreamPipe significantly reduces development time (x6.3) and error rates (x5.19), as measured by a novel Error-Free Score (EFS), compared to LLM code-generation methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。