用风险预测优化测试批次，减少性能回归检测的资源消耗。
Risk-Aware Batch Testing for Performance Regression Detection
- 结合代码风险评分与自适应批量策略，动态安排测试顺序。
- 测试次数减少32.4%，反馈时间缩短3.8%，关键问题定位更快。
- 适合大规模CI系统中的自动化测试优化，尤其关注成本与效率平衡。
性能回归测试在大规模持续集成（CI）系统中至关重要，但每次提交都运行完整测试套件成本过高。现有预测模型与传统批处理策略各有局限：前者少被集成进CI决策，后者忽略提交差异性。本文提出风险感知框架，融合机器学习的提交风险评估与自适应批量方法。以Mozilla Firefox为案例，构建了与Autoland时间对齐的人工确认回归数据集，微调ModernBERT、CodeBERT和LLaMA-3.1变体，用CodeBERT实现最高0.694 ROC-AUC。风险评分驱动多种风险感知批处理策略，包括风险加权优先批处理（RAPB-la）和风险自适应流批处理，在数千次历史提交上模拟验证。最佳配置（RAPB-la）相比生产基准实现帕累托改进：总测试执行减少32.4%，平均反馈时间下降3.8%，平均定位时间维持相近水平，最大定位时间降低26.2%，按模型估算年节省基础设施成本约49.1万美元。结果表明，风险感知批处理可在降低资源消耗的同时提升诊断及时性。为支持可复现性，我们发布完整复现包，含所有数据集、微调流程及算法实现。
原文摘要 · Abstract (English)
Performance regression testing is essential in large-scale continuous-integration (CI) systems, yet executing full performance suites for every commit is prohibitively expensive. Prior work on performance regression prediction and batch testing has shown independent benefits, but each faces practical limitations: predictive models are rarely integrated into CI decision-making, and conventional batching strategies ignore commit-level heterogeneity. We unify these strands by introducing a risk-aware framework that integrates machine-learned commit risk with adaptive batching. Using Mozilla Firefox as a case study, we construct a production-derived dataset of human-confirmed regressions aligned chronologically with Autoland, and fine-tune ModernBERT, CodeBERT, and LLaMA-3.1 variants to estimate commit-level performance regression risk, achieving up to 0.694 ROC-AUC with CodeBERT. The risk scores drive a family of risk-aware batching strategies, including Risk-Aged Priority Batching and Risk-Adaptive Stream Batching, evaluated through realistic CI simulations. Across thousands of historical Firefox commits, our best overall configuration, Risk-Aged Priority Batching with linear aggregation (RAPB-la), yields a Pareto improvement over Mozilla's production-inspired baseline. RAPB-la reduces total test executions by 32.4%, decreases mean feedback time by 3.8%, maintains mean time-to-culprit at approximately the baseline level, reduces maximum time-to-culprit by 26.2%, and corresponds to an estimated annual infrastructure cost savings of approximately $491K under our cost model. These results demonstrate that risk-aware batch testing can reduce CI resource consumption while improving diagnostic timeliness. To support reproducibility and future research, we release a complete replication package containing all datasets, fine-tuning pipelines, and implementations of our batching algorithms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。