用机器学习自动发现BDD测试中可重构的重复步骤片段。
Given, When, Then, Again: Mining Subscenario Refactoring Candidates in Behaviour-Driven Test Suites with ML Classifiers and LLM-Judge Baselines
- 通过语义聚类识别相似的测试步骤序列，找出可重构片段。
- 模型准确率达F1=0.891，优于规则基线和大模型判断。
- 发现75%的测试用例有可提取的背景片段，适合自动化维护者使用。
行为驱动开发（BDD）测试套件中存在大量重复的步骤子序列。已有三种重构模式：文件内背景、仓库内可复用场景调用、跨组织共享高级步骤，但缺乏自动化方法判断哪些重复序列值得提取及适用哪种模式。本研究对339个仓库、276个上游项目组成的Gherkin语料库进行分析，采用滑动窗口（长度L在[2,18]间）提取所有连续步骤序列，利用SBERT/UMAP/HDBSCAN进行语义聚类，识别语义等价片段。三名作者基于书面标准标注了200个代表性片段，训练出一个基于XGBoost的提取价值分类器，并与规则基线及两个开源大语言模型（LLM）裁判对比。结果表明，共发现5,382,249个片段，归并为692,020个重复模式；分类器在五折交叉验证中达到0.891的外样本F1值（95%置信区间[0.852, 0.927]），显著优于规则基线（F1=0.836，p=0.017）和更优的LLM裁判（F1=0.728，p=1.5e-4）。75.0%、59.5%和11.7%的测试场景分别含有文件内背景、仓库内可复用场景、跨组织共享步骤候选，且结果在阈值变化下稳定。研究产出涵盖全生态重构候选普查，相关代码、模型、标注数据及评估标准已开源（Apache-2.0）。
原文摘要 · Abstract (English)
Context. Behaviour-Driven Development (BDD) test suites accumulate duplicated step subsequences. Three published refactoring patterns are available (within-file Background, within-repo reusable-scenario invocation, cross-organisational shared higher-level step), but no prior work automates which recurring subsequences are worth extracting or which mechanism applies. Objective. Rank recurring step subsequences ("slices") by refactoring suitability (extraction-worthy), pre-map each to one of the three patterns, and quantify prevalence across the public BDD ecosystem. Method. Every contiguous L-step window (L in [2, 18]) in a 339-repository / 276-upstream-owner Gherkin corpus is keyed by paraphrase-robust cluster identifiers and counted under three scopes. SBERT / UMAP / HDBSCAN clustering recovers paraphrase-equivalent slices. Three authors label a stratified 200-slice pool against a written rubric. An XGBoost extraction-worthy classifier trained under 5-fold cross-validation is compared with a tuned rule baseline and two open-weight Large Language Model (LLM) judges. Results. The miner produces 5,382,249 slices collapsing to 692,020 recurring patterns. Three-author Fleiss' kappa = 0.56 (extraction-worthy) and 0.79 (mechanism). The classifier reaches out-of-fold F1 = 0.891 (95% CI [0.852, 0.927]), outperforming both the rule baseline (F1 = 0.836, p = 0.017) and the better LLM judge (F1 = 0.728, p = 1.5e-4). 75.0%, 59.5%, and 11.7% of scenarios carry a within-file Background, within-repo reusable-scenario, and cross-organisational shared-step candidate, respectively; the figures are stable under a sweep of the classifier decision threshold. Conclusion. Paraphrase-robust subscenario discovery yields a corpus-wide census of BDD refactoring candidates; pipeline, classifier predictions, labelled pool, and rubric are released under Apache-2.0.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。