用小模型解析代码，自动识别机器学习流程结构
Using Small Language Models to Reverse-Engineer Machine Learning Pipelines Structures
- 用小语言模型分析代码，识别数据科学中的机器学习流程
- 在多个基准上表现优于传统方法，准确率提升显著
- 适合研究数据科学实践或工具开发的开发者和学者
背景：从源代码中提取机器学习（ML）流程的阶段结构，是深入理解数据科学实践的关键。然而，由于机器学习生态系统的持续演进（如算法、库、数据集的多样化），该任务极具挑战性。现有方法要么依赖不可扩展的手动标注，要么使用无法充分适应领域多样性的机器学习分类器。这些局限凸显了对更灵活、可靠解决方案的需求。目标：我们评估小型语言模型（SLMs）是否能利用其代码理解与分类能力解决上述问题，并进一步推动对数据科学实践的理解。方法：基于两项具有代表性的参考研究开展验证性研究。首先，使用Cochran's Q检验比较多种SLMs性能；最佳模型随后通过两个独立的McNemar's检验与参考研究对比。此外，通过额外的Cochran's Q检验分析不同分类体系定义对性能的影响。最后，采用Pearson卡方检验进行拟合优度分析，将本研究所得数据科学实践洞察与先前研究结果进行对比。
原文摘要 · Abstract (English)
Background: Extracting the stages that structure Machine Learning (ML) pipelines from source code is key for gaining a deeper understanding of data science practices. However, the diversity caused by the constant evolution of the ML ecosystem (e.g., algorithms, libraries, datasets) makes this task challenging. Existing approaches either depend on non-scalable, manual labeling, or on ML classifiers that do not properly support the diversity of the domain. These limitations highlight the need for more flexible and reliable solutions. Objective: We evaluate whether Small Language Models (SLMs) can leverage their code understanding and classification abilities to address these limitations, and subsequently how they can advance our understanding of data science practices. Method: We conduct a confirmatory study based on two reference works selected for their relevance regarding current state-of-the-art's limitations. First, we compare several SLMs using Cochran's Q test. The best-performing model is then evaluated against the reference studies using two distinct McNemar's tests. We further analyze how variations in taxonomy definitions affect performance through an additional Cochran's Q test. Finally, a goodness-of-fit analysis is conducted using Pearson's chi-squared tests to compare our insights on data science practices with those from prior studies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。