用大模型当裁判评估测试覆盖率，又快又省还靠谱
LLM-as-a-Judge for Scalable Test Coverage Evaluation: Accuracy, Operational Reliability, and Cost
- 设计可落地的评分框架，用大模型自动判断Gherkin测试脚本质量
- 小模型GPT-4o Mini准确率最高，成本仅为大模型的1/78
- 首次系统分析大模型评估的准确性、可靠性与成本三者关系
在大规模软件测试中，自动化评估测试覆盖范围仍存在瓶颈。本文提出生产级的LLM-as-a-Judge（LAJ）框架，基于评分标准对Gherkin验收测试进行结构化评估。在100个专家标注脚本上，针对20种模型配置（包括GPT-4、GPT-5及开源模型）进行5次重复评估，共完成500次测试。引入评估完成率（ECR@1）衡量首次尝试成功率，结果从85.4%到100.0%，并揭示重试带来的实际成本差异。实验表明，较小模型表现更优：GPT-4o Mini达到最佳准确率（6.07 MAAE），可靠率96.6% ECR@1，成本仅$1.01/千次，相比高推理强度的GPT-5降低78倍；而推理投入增加对开源模型反而造成性能下降。整体成本跨度达175倍（$0.45至$78.96/千次）。相关数据集、框架与代码均已开源。
原文摘要 · Abstract (English)
Assessing software test coverage at scale remains a bottleneck in QA pipelines. We present LLM-as-a-Judge (LAJ), a production-ready, rubric-driven framework for evaluating Gherkin acceptance tests with structured JSON outputs. Across 20 model configurations (GPT-4, GPT-5 with varying reasoning effort, and open-weight models) on 100 expert-annotated scripts over 5 runs (500 evaluations), we provide the first comprehensive analysis spanning accuracy, operational reliability, and cost. We introduce the Evaluation Completion Rate (ECR@1) to quantify first-attempt success, revealing reliability from 85.4% to 100.0% with material cost implications via retries. Results show that smaller models can outperform larger ones: GPT-4o Mini attains the best accuracy (6.07 MAAE), high reliability (96.6% ECR@1), and low cost ($1.01 per 1K), yielding a 78x cost reduction vs. GPT-5 (high reasoning) while improving accuracy. Reasoning effort is model-family dependent: GPT-5 benefits from increased reasoning (with predictable accuracy-cost tradeoffs), whereas open-weight models degrade across all dimensions as reasoning increases. Overall, cost spans 175x ($0.45-$78.96 per 1K). We release the dataset, framework, and code to support reproducibility and deployment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。