用本地大模型让普通人秒级查询海量数据并自动生成安全SQL。
StreamLink: Large-Language-Model Driven Distributed Data Engineering System
- 用领域适配的本地大模型理解自然语言指令，自动生成SQL。
- 生成SQL准确率超基线10%以上，百亿级数据查询仅需数秒。
- 兼顾隐私与安全，适合非技术人员快速访问复杂数据库。
大型语言模型（LLMs）在自然语言理解方面表现出色，为创新应用开辟了新路径。我们提出StreamLink——一个基于分布式框架（如Apache Spark和Hadoop）的LLM驱动型数据系统，旨在提升数据工程任务的效率与易用性。其核心设计原则是尊重用户数据隐私，采用本地微调的LLM而非公开AI服务（如ChatGPT）。通过领域适配的LLM，系统能更好理解用户在不同场景下的自然语言查询，并简化结构化查询语言（SQL）的生成过程。此外，系统集成基于LLM的语法与安全校验器，确保生成查询的可靠性和安全性。实验表明，该系统在生成SQL的执行准确率上超过基线方法10%以上，且可在数秒内从数亿条数据中定位用户关注项，展现出生成式LLM与分布式数据处理融合的巨大潜力。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have shown remarkable proficiency in natural language understanding (NLU), opening doors for innovative applications. We introduce StreamLink - an LLM-driven distributed data system designed to improve the efficiency and accessibility of data engineering tasks. We build StreamLink on top of distributed frameworks such as Apache Spark and Hadoop to handle large data at scale. One of the important design philosophies of StreamLink is to respect user data privacy by utilizing local fine-tuned LLMs instead of a public AI service like ChatGPT. With help from domain-adapted LLMs, we can improve our system's understanding of natural language queries from users in various scenarios and simplify the procedure of generating database queries like the Structured Query Language (SQL) for information processing. We also incorporate LLM-based syntax and security checkers to guarantee the reliability and safety of each generated query. StreamLink illustrates the potential of merging generative LLMs with distributed data processing for comprehensive and user-centric data engineering. With this architecture, we allow users to interact with complex database systems at different scales in a user-friendly and security-ensured manner, where the SQL generation reaches over 10\% of execution accuracy compared to baseline methods, and allow users to find the most concerned item from hundreds of millions of items within a few seconds using natural language.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。