用大模型实现数据湖中语义化表搜索与层级目录生成
LEDD: Large Language Model-Empowered Data Discovery in Data Lakes
- 基于大模型构建端到端数据发现系统，支持自然语言查询表
- 可返回语义相关表并生成带语义的层级全局目录
- 提供简单接口，适合数据工程师和研究者扩展使用
随着数据湖中数据集不断增长，数据发现已成为数据管理领域的一大挑战，尤其在表的语义搜索和全局目录的层次化生成方面。尽管大语言模型（LLMs）有助于处理数据语义，但如何构建一个完整利用LLMs的端到端系统仍面临挑战。本演示提出LEDD，一种具有可扩展架构的端到端系统，利用大模型为数据湖提供带有语义的层级全局目录和语义表搜索功能。具体而言，LEDD可根据自然语言描述返回语义相关的表，该能力使其成为下游任务（如模型训练、文本到SQL的模式链接）的理想基础。LEDD还提供简洁的Python接口，便于算法的扩展与替换。
原文摘要 · Abstract (English)
Data discovery in data lakes with ever increasing datasets has long been recognized as a big challenge in the realm of data management, especially for semantic search of and hierarchical global catalog generation of tables. While large language models (LLMs) facilitate the processing of data semantics, challenges remain in architecting an end-to-end system that comprehensively exploits LLMs for the two semantics-related tasks. In this demo, we propose LEDD, an end-to-end system with an extensible architecture that leverages LLMs to provide hierarchical global catalogs with semantic meanings and semantic table search for data lakes. Specifically, LEDD can return semantically related tables based on natural-language specification. These features make LEDD an ideal foundation for downstream tasks such as model training and schema linking for text-to-SQL tasks. LEDD also provides a simple Python interface to facilitate the extension and the replacement of data discovery algorithms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。