arXiv:2501.09745cs.SEcs.CL2025-01被引 3

用大模型预测代码编辑,首次构建了真实机器学习笔记本的修改数据集。

Suggesting Code Edits in Interactive Machine Learning Notebooks Using Large Language Models

  • 基于GitHub上792个仓库的4.8万次编辑,构建首个细粒度笔记本修改数据集。
  • 平均每次修改仅166行代码,说明编辑高度局部化,但模型准确率仍很低。
  • 强调上下文信息对提升模型性能至关重要,适合研究代码生成与维护的开发者。

机器学习开发者常使用Jupyter笔记本编写数据处理和模型训练代码。尽管这类笔记本便于交互式开发与观察输出,但因文件长度和复杂性,维护工作(如添加新功能或修复缺陷)仍具挑战性。目前尚无针对Jupyter笔记本开发者编辑行为的基准数据集。为此,我们首次构建了一个包含48,398次编辑的数据集,源自792个机器学习仓库在GitHub上的20,095次版本修订。该数据集记录了细胞级和行级的细粒度修改,为理解真实世界中的机器学习工作流维护模式提供了基础。我们发现,这些笔记本的编辑高度局部化,平均每次修改仅涉及166行代码。尽管大模型在代码编辑任务中表现优于小模型,但在经过微调后,所有模型在该数据集上的准确率依然偏低,揭示了真实场景下机器学习维护任务的复杂性。研究强调了上下文信息在提升模型性能中的关键作用,并指出了改进大语言模型在工程化机器学习代码能力方面的潜在方向。

原文摘要 · Abstract (English)

Machine learning developers frequently use interactive computational notebooks, such as Jupyter notebooks, to host code for data processing and model training. Jupyter notebooks provide a convenient tool for writing machine learning pipelines and interactively observing outputs, however, maintaining Jupyter notebooks, e.g., to add new features or fix bugs, can be challenging due to the length and complexity of the notebooks. Moreover, there is no existing benchmark related to developer edits on Jupyter notebooks. To address this, we present the first dataset of 48,398 Jupyter notebook edits derived from 20,095 revisions of 792 machine learning repositories on GitHub, and perform the first study of the using LLMs to predict code edits in Jupyter notebooks. Our dataset captures granular details of cell-level and line-level modifications, offering a foundation for understanding real-world maintenance patterns in machine learning workflows. We observed that the edits on Jupyter notebooks are highly localized, with changes averaging only 166 lines of code in repositories. While larger models outperform smaller counterparts in code editing, all models have low accuracy on our dataset even after finetuning, demonstrating the complexity of real-world machine learning maintenance tasks. Our findings emphasize the critical role of contextual information in improving model performance and point toward promising avenues for advancing large language models' capabilities in engineering machine learning code.

代码生成大模型Jupyter维护

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。