开源框架评估本地大模型在纵向数据准备中的表现
Agentic coding without the cloud: evaluating open-weight large language models on longitudinal data preparation tasks
- 构建本地可运行的AI代理评估框架，支持敏感数据不出本地
- 31-35B参数模型在20个任务中平均完成率达87.9%
- 适合受数据治理限制的研究场景使用
大型语言模型（LLM）和智能体现已成为代码开发的常用工具，但通常需将数据上传至第三方云服务。在涉及个人数据的研究中，这一做法受限于数据治理要求。本地部署的开源权重模型可避免数据外流。本文提出一个开源框架，用于评估基于开源权重LLM的AI代理在纵向队列研究中最常见的瓶颈——数据准备中的表现。该框架包含：经验证的真实数据集（来自英国队列研究的六轮数据清洗脚本）、任务定义（如类别标准化、多波次合并）以及自动化评估机制，用于检验LLM生成的R代码和输出数据。我们在不同消费级硬件上对多种模型进行基准测试，涵盖20项数据准备任务（生成102个变量）。当前最先进的31-35B参数模型几乎达到性能上限（平均任务完成率87.9%）。结果表明，本地运行的开源模型在受治理约束的研究环境中具备实现AI辅助数据准备的可行性。框架已公开：https://github.com/UCL-ARC/RRBench。
原文摘要 · Abstract (English)
Large language models (LLMs) and agents are now widely used tools in code development, with data typically sent to third-party cloud-based models. Their adoption in research using personal data is constrained by governance requirements that typically prohibit data transmission to external services. Locally deployable open-weight models offer an alternative since sensitive data never leave the local environment. We introduce an open-source framework for evaluating the efficacy of AI agents powered by open-weight LLMs on one of the most persistent bottlenecks in research on longitudinal population studies: data preparation. The framework comprises: a curated ground-truth dataset (cleaning scripts preparing six sweeps of data from a British cohort study), task definitions encompassing tasks such as category harmonization and multi-wave merging, and automated routines for evaluating the LLM-produced R code and outputted data. We benchmark LLMs across the (consumer grade) deployment spectrum to assess their efficacy in 20 data preparation tasks (creation of 102 variables). Current state-of-the-art, 31-35B parameter models almost saturated our benchmark ('average task completion' up to 87.9%). The performance of open-weight LLMs running on consumer-grade hardware shows promise of a viable path toward AI-assisted data preparation in governance-restricted research settings. Our framework is publicly available at: https://github.com/UCL-ARC/RRBench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。