arXiv:2607.22165cs.DBcs.AI2026-07

为大模型数据库代理设计了高保真评测基准,真实还原生产环境挑战。

DBA-Bench: A Production-Fidelity Benchmark for LLM-Based Database Operations Agents

论文配图:DBA-Bench: A Production-Fidelity Benchmark for LLM-Based Database Operations Agents
图 1 · 摘自论文原文
  • 构建带实时负载的可复现数据库环境,模拟真实运维场景。
  • 自动化测试中安全修复率仅12.4%,人类专家达93.4%。
  • 覆盖7类任务、106个故障场景,区分难易程度便于精准评估。

基于大模型的数据库代理虽具潜力,但因任务范围、测试环境与评估指标不一,难以比较。本文指出评估与生产运维间的四大差距:真实环境保真度(多轮读写交互)、观测空间规模与复杂性(跨数千时间序列、业务日志及并发活动的因果诊断)、解决方案开放性(多种修复策略及其权衡)以及场景复杂性与覆盖度(故障在内部机制与操作域间级联)。为此提出DBA-Bench基准,通过生产级保真度、以结果为导向的评估和受控场景复现解决上述问题。该基准采用注入监控的PostgreSQL环境,具备活跃工作负载、持久状态和多源观测;成功定义为在安全约束下实现可测量恢复或故障消除;每次运行前通过特定检查点恢复快照。包含7个任务领域共106个场景,提供基于参考路径诊断深度与环境复杂性的两个公开难度标签。对九组基线模型(含六种基础模型系统、两个基于GPT-5.5的数据库代理及一人类数据库管理员参考)进行848次自动化测试,诊断成功率、结果达成率与安全通过率分别为32.7%、19.6%、12.4%;最佳自动基线安全通过率为17.9%,远低于人类参考的93.4%。安全通过率从易场景的19.6%降至难场景的7.6%,凸显安全端到端修复的艰巨性。

原文摘要 · Abstract (English)

LLM-based database agents show promise, but differing task scopes, testbeds, and metrics hinder comparison. We identify four gaps between evaluation and production operations: live-environment fidelity (multi-turn read-write interaction with a running database); observation-space scale and complexity (causal diagnosis across thousands of time series, business logs, and concurrent activity); solution-space openness (multiple remediations with different operational trade-offs); and scenario complexity and coverage (faults cascading across internal mechanisms and operational domains). We present DBA-Bench, a benchmark addressing these gaps through production fidelity, outcome-first evaluation, and controlled scenario reproducibility. It uses instrumented PostgreSQL environments with active workloads, persistent state, and multi-source observations; defines success by measurable recovery or fault elimination under safety constraints; and restores snapshots with scenario-specific checks before each run. The benchmark contains 106 scenarios across seven task domains, with two public difficulty labels based on reference-path diagnostic depth and environmental complexity. We evaluate nine baseline groups, including six foundation-model systems, two GPT-5.5-backed database agents, and a Human DBA reference. Across 848 automated runs, Diagnosis, Outcome, and Safe Pass rates are 32.7%, 19.6%, and 12.4%; the best automated baseline reaches 17.9% Safe Pass versus 93.4% for the Human DBA reference. Automated Safe Pass falls from 19.6% on Easy scenarios to 7.6% on Hard scenarios, underscoring the difficulty of safe end-to-end remediation.

数据库大模型评测AI运维基准测试

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。