评测大模型在统计问题建模中的表现，发现现有模型能力有限。
Benchmarking Language Models for Statistical Problem Formulation
- 将统计问题建模分解为分类与变量识别两个子任务
- 最佳模型零样本准确率仅72.0%，变量重合度63.2%
- 适合关注模型在真实数据分析中推理能力的研究者
大语言模型在统计与数据科学工作中日益作为助手使用，但现有评估多假设分析目标已明确。现实中，用户常以非正式目标和异构数据出发，需模型自行判断隐含的统计任务及相关数据。本文首次将此上游步骤形式化为统计问题建模，并分解为统计问题分类与变量识别及角色分配两子任务。我们构建了StatFormBench基准，基于五本跨领域统计教材与一个数据科学案例库，涵盖20类粗粒度与85类细粒度统计问题类别，共1,013个样本。在14个开源与闭源大模型上测试，最优零样本模型在细粒度分类上准确率达72.0%，变量集重合度为63.2%。无模型在两项任务上均表现最佳，增强提示策略仅带来有限或不一致提升。基准数据已发布于Hugging Face，评估代码同步开源。
原文摘要 · Abstract (English)
Large language models (LLMs) are increasingly used as assistants for statistical and data science work, yet existing evaluations largely assume the analysis target is already specified. In practice, users arrive with informal goals and heterogeneous data, leaving the model to decide what statistical task is implied and which data are relevant. We first formalize this upstream step as Statistical Problem Formulation and decompose it into two subtasks: (1) Statistical Problem Classification and (2) Variable Identification & Role Assignment. We then introduce StatFormBench, a benchmark built from five cross-domain statistics textbooks and a data science case library, covering diverse problem types, data representations, and scenario styles. It contains 1,013 samples spanning 20 coarse-grained and 85 fine-grained statistical problem categories. Across 14 open- and closed-source LLMs, the best zero-shot models reach only 72.0 fine-grained classification accuracy and 63.2 variable set overlap. No model performs consistently best across the two subtasks, while enhanced prompting strategies yield only limited or inconsistent gains. We release the benchmark data on Hugging Face at https://huggingface.co/datasets/THU-CongLab/StatFormBench and the evaluation code on GitHub at https://github.com/THU-CongLab/StatFormBench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。