首个覆盖数据库全生命周期的LLM评估基准，揭示专用模型在维护阶段能力严重退化。
Evaluating LLMs in Database Scenarios: A Lifecycle Benchmark for Assessing Their Potential in Core Database Tasks
- 构建覆盖设计、实现、运维等五个阶段的全流程评估框架
- 发现通用模型在各阶段表现均衡，专用模型在非编码任务中出现灾难性遗忘
- 提出渐进式文本转SQL方法，模拟人类分步推理过程
大型语言模型正在重塑数据库交互方式，从简单的文本转SQL演变为自主数据库管理员。然而现有评估基准仍过度聚焦于文本转SQL任务，忽视了从初始模式设计到部署后维护的完整数据库生命周期。这种片面评估无法反映真实数据库管理所需的核心能力。为此，我们提出DBLifeBench，首个覆盖设计、实施、运行、调试和维护五大关键阶段的评估基准。针对自然语言模糊性与复杂SQL逻辑之间的认知错配，我们引入一种基于结构化推理图的渐进式文本转SQL新任务，以模拟人类迭代式问题求解过程。大规模实验揭示关键洞见：尽管通用模型在各阶段表现均衡，但专用文本转SQL模型在设计与维护等非编码阶段存在严重能力退化。DBLifeBench为评估与构建真正的全栈数据库智能奠定了基础。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are transforming database interaction paradigms, evolving from simple query translators to autonomous database administrators (DBAs). However, current evaluation benchmarks remain disproportionately fixated on Text-to-SQL tasks, neglecting the holistic Database Lifecycle-from initial schema design to post-deployment maintenance. This narrow focus fails to capture the diverse capabilities required for real-world database management. To bridge this gap, we introduce DBLifeBench, the first benchmark to evaluate LLMs across five critical lifecycle phases: Design, Implementation, Operation, Debugging, and Maintenance. Furthermore, addressing the cognitive mismatch between ambiguous natural language and complex SQL logic, we propose Progressive-Text2SQL, a novel task utilizing structured reasoning graphs to mimic human iterative problem-solving. Our extensive evaluation reveals a critical insight: while general-purpose models demonstrate balanced performance, specialized Text-to-SQL models suffer from ``catastrophic forgetting'' in non-coding phases like design and maintenance. DBLifeBench serves as a foundational step toward evaluating and building true full-stack database intelligence.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。