首个面向板球分析的多语言文本转SQL评测基准，揭示大模型在体育数据上的理解短板。
CricBench: A Multilingual Benchmark for Evaluating LLMs in Cricket Analytics
- 构建涵盖四种板球赛制的多语言数据集，评估大模型生成精确查询语句能力。
- 所有模型在奥迪赛制上表现为零准确率，且语义正确率普遍低于29%。
- 发现语法正确与实际意义正确的巨大差距，适合体育数据分析研究者参考。
板球是全球第二受欢迎的运动，拥有数十亿粉丝，他们渴望获得常规网络搜索无法提供的高级统计洞察。尽管大语言模型在文本转SQL任务上已取得显著进展，但其在体育分析中处理领域特异性与多语言需求的能力仍鲜有探索。我们提出CricBench，一个评估大模型在板球数据上内在SQL生成能力的基准套件，覆盖测试赛、ODI、T20I和IPL四种赛制。我们构建了包含2,654个评估实例的黄金标准数据集，涵盖英语、印地语、旁遮普语和泰卢固语四种语言。评估了七种模型：GPT-5 Mini、Claude Sonnet 4、DeepSeek R1和V3、Qwen 235B、Llama 3.1和Gemma 2，均采用仅模式提示（schema-only prompting）。无单一模型在所有赛制上领先：GPT-5 Mini在测试赛中表现最佳（12.4% DMA），Qwen 235B在IPL（28.7%）和T20I（17.5%）中领先，所有模型在困难的ODI查询上得分均为0%。所有模型均表现出严重语法正确性（>98%执行准确率）与语义正确性（<29% DMA）之间的脱节，与BIRD相比存在37-55个百分点的领域差距。据我们所知，CricBench是首个针对板球分析的文本转SQL基准。
原文摘要 · Abstract (English)
Cricket is the second most popular sport worldwide, with billions of fans seeking advanced statistical insights unavailable through standard web searches. Although LLMs have advanced significantly in Text-to-SQL tasks, their capability to handle domain-specific nuances and multilingual requirements in sports analytics remains under-explored. We present CricBench, a benchmark suite evaluating the intrinsic SQL generation abilities of LLMs on cricket data across four formats: Test, ODI, T20I, and IPL. We curate a Gold-Standard dataset of 2,654 evaluation instances across four languages (English, Hindi, Punjabi, and Telugu). We evaluate seven models, GPT-5 Mini, Claude Sonnet 4, DeepSeek R1 and V3, Qwen 235B, Llama 3.1, and Gemma 2, using schema-only prompting. No single model dominates across all formats: GPT-5 Mini leads on Test cricket (12.4% DMA), Qwen 235B leads on IPL (28.7%) and T20I (17.5%), and all models score 0% on hard ODI queries. All models show a stark disconnect between syntactic validity (>98% execution accuracy) and semantic correctness (<29% DMA), with a domain gap of 37-55 percentage points versus BIRD. To our knowledge, CricBench is the first Text-to-SQL benchmark for cricket analytics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。