首个多轮文本转SQL基准测试，揭示记忆机制对复杂查询的关键影响。
Memory Architectures for Multi-Turn Text-to-SQL: A Benchmark and Empirical Study
- 构建300会话1400轮的多轮Text-to-SQL基准，含精确标注的内存依赖信息。
- 发现所有模型在第3轮后执行准确率归零，且记忆架构复杂度不提升性能。
- 揭示大模型生成错误具有单一模式特征，适合企业级分析系统优化参考。
多轮文本转SQL是企业数据分析的核心，但现有评估仍以单轮为主。本文提出EnterpriseMem-Bench，一个由三个企业领域（BIRD金融、SEC EDGAR、Northwind）程序化构建的多轮文本转SQL基准，包含300个会话共1,400轮，具备确定性真值和逐轮记忆关键性标注。我们在五种记忆条件下评估五款前沿模型（GPT-5 mini、GPT-5.2、Claude Sonnet 4.5、Sonnet 4.6、Opus 4.6），通过三向消融分离工作记忆窗口大小、情景检索与语义增强的影响。所有Claude模型均开启扩展推理以保证与GPT模型的可比性。引入每轮诊断指标Memory Benefit Score（MBS）。四项发现：(1) 无状态多轮模型在所有模型中，第3轮执行准确率降至0；(2) 记忆架构复杂度非单调提升准确率，工作记忆主导，额外组件在不同模型和数据集上带来+14至-16个百分点的波动；(3) Claude Sonnet 4.6在SEC EDGAR上表现低于Sonnet 4.5达17-33个百分点，该代际退化在推理开启后依然存在；(4) 开启推理后，Claude的错误分布呈现单峰——所有非正确轮次均为结果错误。论文发布基准、评测代理及代码。
原文摘要 · Abstract (English)
Multi-turn Text-to-SQL is central to enterprise analytics yet remains predominantly evaluated in single-turn settings. We introduce EnterpriseMem-Bench, a multi-turn Text-to-SQL benchmark of 300 sessions and 1,400 turns built programmatically from three enterprise domains (BIRD financial, SEC EDGAR, Northwind), with deterministic ground truth and per-turn memory-critical annotation. We evaluate five frontier models -- GPT-5 mini, GPT-5.2, Claude Sonnet 4.5, Sonnet 4.6, and Opus 4.6 -- across five memory conditions enabling a three-way ablation isolating working-memory window size, episodic retrieval, and semantic augmentation as independent effects. All Claude models are evaluated with extended thinking enabled to maintain parity with GPT reasoning models. We introduce the Memory Benefit Score (MBS) as a per-turn diagnostic metric. Four findings emerge: (1) stateless multi-turn Text-to-SQL collapses to zero execution accuracy by Turn 3 across all five models, even under reasoning; (2) memory-architecture complexity does not monotonically improve accuracy -- working memory dominates, and additional components produce model- and dataset-dependent effects from +14 to -16 percentage points; (3) Claude Sonnet 4.6 underperforms Sonnet 4.5 by 17-33pp on SEC EDGAR across conditions, a generational regression persisting under reasoning; (4) under reasoning, Claude error distributions become mono-modal -- every non-correct turn is a wrong-result error. We release the benchmark, agent, and evaluation code.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。