为软件工程定制开源模型分类,助力高效选型与复用。
Cataloguing Hugging Face Models to Software Engineering Activities: Automation and Findings
- 构建147类软件工程任务的分类体系,自动标注Hugging Face中的模型。
- 2205个模型中代码生成占主导,但仅9.6%有基准测试结果。
- 适合关注模型选型、自动化开发的工程师与研究者参考。
背景:开源预训练模型(PTMs)为多种机器学习任务提供丰富资源,但缺乏针对软件工程(SE)需求的分类体系,难以可靠识别和复用模型。目标:为此,我们构建涵盖147个SE任务的分类体系,并对流行的开源机器学习库Hugging Face(HF)中的模型进行面向SE的分类。方法:采用五阶段数据挖掘流程:(i) 从文献中提取SE任务;(ii) 通过HF API获取模型卡片描述、元数据及关联arXiv论文摘要;(iii) 文本处理确保一致性;(iv) 两阶段相关性验证,结合人工与大语言模型辅助,经五次试点研究和泛化测试;(v) 数据分析。最终生成2,205个经筛选的SE相关PTM目录。结果:多数模型聚焦代码生成与编码任务,侧重实现阶段而非早期或后期开发。在机器学习任务上,文本生成占主导。值得注意的是,自2023年第二季度以来,SE相关模型数量显著增长,但评估仍不足:仅有9.6%报告基准测试结果,且多数得分低于50%。结论:目录揭示了文档与透明度缺失、软件开发生命周期各阶段不平衡等问题,为自动化场景下的模型采样与选择提供了基础。
原文摘要 · Abstract (English)
Context: Open-source Pre-Trained Models (PTMs) provide extensive resources for various Machine Learning (ML) tasks, yet these resources lack a classification tailored to Software Engineering (SE) needs to support the reliable identification and reuse of models for SE. Objective: To address this gap, we derive a taxonomy encompassing 147 SE tasks and apply an SE-oriented classification to PTMs in a popular open-source ML repository, Hugging Face (HF). Method: Our repository mining study followed a five-phase pipeline: (i) identification SE tasks from the literature; (ii) collection of PTM data from the HF API, including model card descriptions and metadata, and the abstracts of the associated arXiv papers; (iii) text processing to ensure consistency; (iv) a two-phase validation of SE relevance, involving humans and LLM assistance, supported by five pilot studies with human annotators and a generalization test; (v) and data analysis. This process yielded a curated catalogue of 2,205 SE PTMs. Results: We find that most SE PTMs target code generation and coding, emphasizing implementation over early or late development stages. In terms of ML tasks, text generation dominates within SE PTMs. Notably, the number of SE PTMs has increased markedly since 2023 Q2, while evaluation remains limited: only 9.6% report benchmark results, mostly scoring below 50%. Conclusions: Our catalogue reveals documentation and transparency gaps, highlights imbalances across SDLC phases, and provides a foundation for automated SE scenarios, such as the sampling and selection of suitable PTMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。