用大模型自动完成数据集成，省时省钱还效果不差
Automatic End-to-End Data Integration using Large Language Models
- 用GPT-5.2自动生成模式映射、值映射、训练数据等全流程配置
- 在游戏、音乐、企业数据集成中表现接近甚至优于人工设计管道
- 每案例成本约10美元，仅为人工成本的一小部分
设计数据集成管道通常需要数据工程师大量手动配置组件和标注训练数据。尽管大语言模型在集成流程的个别步骤中已展现出潜力，但其在端到端集成管道中完全替代人工输入的可能性尚未被研究。为此，我们提出一个基于GPT-5.2的自动化数据集成管道，可自动生成适配具体用例所需的全部产物：模式映射、数据归一化的值映射、实体匹配的训练数据，以及用于选择冲突解决启发式策略的验证数据。我们在三个案例研究中对比了该模型驱动管道与人工设计管道的性能，涵盖游戏、音乐和公司相关数据的集成。实验表明，该模型管道在某些任务上甚至优于人工管道，整体产出的数据集在规模和密度上与人工管道相当。使用大模型配置管道的成本约为每案例10美元，远低于人工所需成本。
原文摘要 · Abstract (English)
Designing data integration pipelines typically requires substantial manual effort from data engineers to configure pipeline components and label training data. While LLMs have shown promise in handling individual steps of the integration process, their potential to replace all human input across end-to-end data integration pipelines has not been investigated. As a step toward exploring this potential, we present an automatic data integration pipeline that uses GPT-5.2 to generate all artifacts required to adapt the pipeline to specific use cases. These artifacts are schema mappings, value mappings for data normalization, training data for entity matching, and validation data for selecting conflict resolution heuristics in data fusion. We compare the performance of this LLM-based pipeline to the performance of human-designed pipelines along three case studies requiring the integration of video game, music, and company related data. Our experiments show that the LLM-based pipeline is able to produce similar results, for some tasks even better results, as the human-designed pipelines. End-to-end, the human and the LLM pipelines produce integrated datasets of comparable size and density. Having the LLM configure the pipelines costs approximately \$10 per case study, which represents only a small fraction of the cost of having human data engineers perform the same tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。