AIRepr自动评估LLM数据科学分析的可复现性,提升结果准确性和可靠性。
AIRepr: An Analyst-Inspector Framework for Evaluating Reproducibility of LLMs in Data Science
- 基于统计原理构建分析-检查框架,自动评估代码生成逻辑
- 15对LLM在1032个任务上验证,可复现性提升带来准确率显著提高
- 提出新提示策略,适合需透明可信数据决策的科研与工程团队
大型语言模型(LLMs)正被广泛用于通过生成可执行代码来自动化数据科学分析。然而,数据科学任务常存在多种统计上有效的解决方案,如不同的建模策略,因此理解分析背后的推理过程比仅关注结果更为关键。尽管人工审查生成代码有助于确保统计合理性,但耗时且依赖专业知识。更可扩展的方法是评估底层工作流——指导代码生成的逻辑计划。然而,目前尚不清楚如何评估LLM生成的工作流是否支持可复现实现。为此,我们提出AIRepr,一种分析师-检查员框架,用于自动评估并改进LLM生成的数据分析工作流的可复现性。该框架基于统计原则,支持可扩展的自动化评估。我们引入两种新颖的可复现性增强提示策略,并在15组分析师-检查员LLM组合及三个公开基准数据集的1,032个任务上与标准提示进行对比。结果表明,具有更高可复现性的工作流也产生更准确的分析,且可复现性增强提示显著提升了两项指标。本工作为数据科学中透明、可靠且高效的人机协作奠定了基础。代码已公开。
原文摘要 · Abstract (English)
Large language models (LLMs) are increasingly used to automate data analysis through executable code generation. Yet, data science tasks often admit multiple statistically valid solutions, e.g. different modeling strategies, making it critical to understand the reasoning behind analyses, not just their outcomes. While manual review of LLM-generated code can help ensure statistical soundness, it is labor-intensive and requires expertise. A more scalable approach is to evaluate the underlying workflows-the logical plans guiding code generation. However, it remains unclear how to assess whether an LLM-generated workflow supports reproducible implementations. To address this, we present AIRepr, an Analyst-Inspector framework for automatically evaluating and improving the reproducibility of LLM-generated data analysis workflows. Our framework is grounded in statistical principles and supports scalable, automated assessment. We introduce two novel reproducibility-enhancing prompting strategies and benchmark them against standard prompting across 15 analyst-inspector LLM pairs and 1,032 tasks from three public benchmarks. Our findings show that workflows with higher reproducibility also yield more accurate analyses, and that reproducibility-enhancing prompts substantially improve both metrics. This work provides a foundation for transparent, reliable, and efficient human-AI collaboration in data science. Our code is publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。