AI可自主构建与运维整个数据栈,实现全生命周期自动化。
Can AI autonomously build, operate, and use the entire data stack?
- 用智能代理分阶段自主管理数据栈各环节
- 推动从局部辅助到全流程自治的范式转变
- 适合关注AI驱动数据系统未来的研究者与工程师
企业数据管理是一项艰巨任务,涵盖数据架构与系统、集成、质量、治理及持续优化。尽管当前AI助手可辅助数据工程师和数据管家等角色,但在全自动化方面仍远未达成。随着AI能力提升,过去因复杂性难以自动化的任务正逐步被攻克,这为实现完全自主的数据资产体系带来契机。本文主张从独立组件的局部应用转向对整个数据生命周期的综合自治。我们探讨智能代理如何自主管理现代数据栈的各个阶段,构建可自我维持的系统,不仅服务于人类用户,也可被AI自身使用。文章分析推动这一变革的动力与机遇,说明代理如何简化数据生命周期,并指出尚待研究的关键问题。我们期望此工作能引发讨论、激发研究、促进协作,共同迈向更自主的数据系统未来。
原文摘要 · Abstract (English)
Enterprise data management is a monumental task. It spans data architecture and systems, integration, quality, governance, and continuous improvement. While AI assistants can help specific persona, such as data engineers and stewards, to navigate and configure the data stack, they fall far short of full automation. However, as AI becomes increasingly capable of tackling tasks that have previously resisted automation due to inherent complexities, we believe there is an imminent opportunity to target fully autonomous data estates. Currently, AI is used in different parts of the data stack, but in this paper, we argue for a paradigm shift from the use of AI in independent data component operations towards a more holistic and autonomous handling of the entire data lifecycle. Towards that end, we explore how each stage of the modern data stack can be autonomously managed by intelligent agents to build self-sufficient systems that can be used not only by human end-users, but also by AI itself. We begin by describing the mounting forces and opportunities that demand this paradigm shift, examine how agents can streamline the data lifecycle, and highlight open questions and areas where additional research is needed. We hope this work will inspire lively debate, stimulate further research, motivate collaborative approaches, and facilitate a more autonomous future for data systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。