评测大模型直接读数据或写SQL答问题的能力
Can Language Models Analyze Data? Evaluating Large Language Models for Question Answering over Datasets
- 让大模型直接读数据文件或生成SQL来回答问题
- 大模型表现优秀，小模型在复杂任务上明显不足
- 适合关注数据问答的开发者与研究者参考
本文评估大语言模型（LLMs）在数据集上回答问题的有效性。实验涵盖两种场景：(a) 直接将数据文件输入模型以回答问题；(b) 根据关系型数据库的表结构生成SQL查询。研究还考察了不同提示策略对性能的影响。测试对象包括先进大模型和资源消耗少的小型语言模型。实验基于两个含不同难度问题的数据集。结果表明大模型表现强劲，而小型模型在复杂任务中存在明显局限。这些发现有助于理解大模型在数据分析中的应用潜力与边界。
原文摘要 · Abstract (English)
This paper investigates the effectiveness of large language models (LLMs) in answering questions over datasets. We examine their performance in two scenarios: (a) directly answering questions given a dataset file as input, and (b) generating SQL queries to answer questions given the schema of a relational database. We also evaluate the impact of different prompting strategies on model performance. The study includes both state-of-the-art LLMs and smaller language models that require fewer resources and operate at lower computational and financial cost. Experiments are conducted on two datasets containing questions of varying difficulty. The results demonstrate the strong performance of large LLMs, while highlighting the limitations of smaller, more cost-efficient models. These findings contribute to a better understanding of how LLMs can be utilized in data analytics tasks and their associated limitations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。