30年软件工程研究中普遍存在统计方法错误,专家也难识别。
A Call for Critically Rethinking and Reforming Data Analysis in Empirical Software Engineering
- 用大模型筛选2.7万篇论文,识别统计方法缺陷
- 33位专家仅部分能发现并修正研究中的统计问题
- 呼吁重审软件工程数据方法,提升研究可信度
经验软件工程(ESE)通过定性与定量研究推动创新,但自2006年达格斯图尔研讨会以来,对实证方法正确应用的担忧持续存在。本文分析了近三十年的软件工程研究,识别统计方法中的常见错误,并评估领域专家检测和纠正这些问题的能力。我们对约2.7万篇实证研究进行了文献调查,利用大语言模型(LLMs)分类统计方法是否恰当;同时选取30项主研究,在包含33位ESE专家的研讨会上评估其发现问题与修复能力。结果显示,主研究中存在显著统计缺陷,且专家在识别与修正方面表现有限,暴露出整个学术群体在方法学上的普遍不足。尽管研究存在局限,其结果揭示了长期存在的‘复制粘贴’现象及不当方法的持续传播,导致不可靠结论泛滥,阻碍了正确统计策略的普及。因此,本文呼吁对经验软件工程中的数据分析进行批判性反思与改革,为建立更严谨的方法框架铺路。
原文摘要 · Abstract (English)
Context: Empirical Software Engineering (ESE) drives innovation in SE through qualitative and quantitative studies. However, concerns about the correct application of empirical methodologies have existed since the 2006 Dagstuhl seminar on SE. Objective: To analyze three decades of SE research, identify mistakes in statistical methods, and evaluate experts' ability to detect and address these issues. Methods: We conducted a literature survey of ~27,000 empirical studies, using LLMs to classify statistical methodologies as adequate or inadequate. Additionally, we selected 30 primary studies and held a workshop with 33 ESE experts to assess their ability to identify and resolve statistical issues. Results: Significant statistical issues were found in the primary studies, and experts showed limited ability to detect and correct these methodological problems, raising concerns about the broader ESE community's proficiency in this area. Conclusions. Despite our study's eventual limitations, its results shed light on recurring issues from promoting information copy-and-paste from past authors' works and the continuous publication of inadequate approaches that promote dubious results and jeopardize the spread of the correct statistical strategies among researchers. Besides, it justifies further investigation into empirical rigor in software engineering to expose these recurring issues and establish a framework for reassessing our field's foundation of statistical methodology application. Therefore, this work calls for critically rethinking and reforming data analysis in empirical software engineering, paving the way for our work soon.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。