arXiv:2601.15728cs.AIcs.SE2026-01

对比文本转Python与文本转SQL,发现后者更依赖隐式逻辑,前者需显式补全上下文。

Benchmarking Text-to-Python against Text-to-SQL: The Impact of Explicit Logic and Ambiguity

  • 通过显式补全领域知识来解决自然语言歧义问题
  • 补全上下文后,文本转Python性能可追平文本转SQL
  • 适合构建需要复杂分析流程的智能数据助手

尽管文本转SQL仍是数据库交互主流,但实际分析任务日益需要通用编程语言如Python或Pandas处理文件数据和复杂工作流。然而,文本转Python在核心数据检索上的可靠性远未得到充分研究。为此,我们提出BIRD-Python基准,系统清理原始数据以减少标注噪声并统一执行语义,建立标准化对比基线。分析表明:SQL依赖声明式结构中的隐式数据库行为,而Python需显式过程逻辑,对用户意图不明确极为敏感。为此我们提出逻辑补全框架(LCF),通过引入潜在领域知识解决歧义。实验显示:(1)性能差异主要源于缺失领域上下文而非代码生成能力不足;(2)当上下文补齐后,文本转Python性能达到与文本转SQL相当水平。这些发现表明,只要能将模糊自然语言输入准确映射为可执行逻辑,Python即可成为分析型代理的可靠基础。资源见https://anonymous.4open.science/r/Bird-Python-43B7/。

原文摘要 · Abstract (English)

While Text-to-SQL remains the dominant approach for database interaction, real-world analytics increasingly require the flexibility of general-purpose programming languages such as Python or Pandas to manage file-based data and complex analytical workflows. Despite this growing need, the reliability of Text-to-Python in core data retrieval remains underexplored relative to the mature SQL ecosystem. To address this gap, we introduce BIRD-Python, a benchmark designed for cross-paradigm evaluation. We systematically refined the original dataset to reduce annotation noise and align execution semantics, thereby establishing a consistent and standardized baseline for comparison. Our analysis reveals a fundamental paradigmatic divergence: whereas SQL leverages implicit DBMS behaviors through its declarative structure, Python requires explicit procedural logic, making it highly sensitive to underspecified user intent. To mitigate this challenge, we propose the Logic Completion Framework (LCF), which resolves ambiguity by incorporating latent domain knowledge into the generation process. Experimental results show that (1) performance differences primarily stem from missing domain context rather than inherent limitations in code generation, and (2) when these gaps are addressed, Text-to-Python achieves performance parity with Text-to-SQL. These findings establish Python as a viable foundation for analytical agents-provided that systems effectively ground ambiguous natural language inputs in executable logical specifications. Resources are available at https://anonymous.4open.science/r/Bird-Python-43B7/.

文本转代码数据分析逻辑补全

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。